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ABSTRACT 


In  the  mid-1980s,  the  assessment  movement  began  to  spread  throughout  academia  as 
colleges  and  universities  created  programs  to  address  the  issues  of  accountability  and 
program  improvement.  A  multitude  of  comprehensive  institution-wide  assessment 
programs  emerged  from  the  movement  which  brought  about  change  on  many  campuses. 
The  purpose  of  this  study  was  to  develop  a  comprehensive  assessment  program  at  an  Air 
Force  professional  military  education  institution,  Air  Command  and  Staff  College  (ACSC), 
based  on  the  perceptions  of  recent  Air  Force  officer  graduates  of  the  program. 

Graduates  (n=395)  were  asked  to  rate  the  quality  of  program  elements  (teaching 
methods  and  program  activities)  and  to  disclose  their  perceived  competence  on  outcome 
variables.  Based  on  the  data  from  a  90-item  questionnaire  titled,  “Student  Perceptions  of 
Program  Effectiveness  Questionnaire,”  the  researcher  analyzed  student  perceptions  on 
three  types  of  variables— inputs  (demographics  and  student  expectations),  environment 
(teaching  methods  and  program  activities),  and  outcomes  (program  goals).  Information 
from  returned  questionnaires  was  collected  and  analyzed  using  descriptive  (means, 
standard  deviations,  and  percentages),  correlational  (cross-tabulations  and  Pearson  “r”s), 
predictive  (multiple  regression)  statistics,  and  qualitative  analysis. 

The  results  of  the  correlational  and  predictive  analyses  show  that  ACSC  graduates 
generally  perceived  their  competencies  on  outcome  variables  and  the  quality  of 
environmental  variables  as  high.  The  most  important  results  emerged  from  the  predictive 
analysis.  After  controlling  for  the  effects  of  inputs,  which  accounted  from  three  percent  of 
the  variance  in  Command  and  Leadership  to  nine  percent  in  Critical  Thinking  outcome 


Leadership  to  fifteen  percent  in  Joint  Campaign  outcome  variables.  These  data  suggest 
that  after  controlling  for  inputs,  respondents  who  rated  Curriculum  as  high  were  more 
likely  to  perceive  their  competence  as  high  on  outcome  variables.  After  Curriculum,  a 
small  amount  of  the  explained  variance  (fi-om  two  percent  in  Air/Space  Power  and  Critical 
Thinking  to  three  percent  in  Joint  Campaign)  came  from  Teaching  Methods.  The  only 
exception  to  this  pattern  was  in  the  Command  and  Leadership  variable  where  Faculty 
became  predictive  along  with  Curriculum,  accounting  for  four  percent  of  the  explained 
variance  over  that  accounted  for  by  Curriculum. 
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CHAPTER  ONE 


INTRODUCTION 

In  the  mid- 1980s  the  term  “assessment”  began  to  appear  in  the  higher  education 
literature  with  increasing  regularity.  Educators  referred  to  the  “movement”  in  their 
articles  and  books  about  assessment  purposes,  concepts,  and  techniques.  Although 
educational  programs  and  the  students  who  enrolled  in  them  have  always  been  subjected 
to  some  type  of  scrutiny,  the  assessment  movement  of  the  1980s  took  on  new  meaning. 
Assessment  was  inexorably  linked  to  another  movement,  the  accountability  movement, 
which  called  for  educators  to  prove  they  were  accomplishing  their  goals  and  doing  it 
eflSciently.  Reports  such  as  William  J.  Bennett’s  monograph.  To  Reclaim  a  Legacy:  A 
Report  on  the  Humanities  in  Higher  Education  (1984)  and  the  Association  of  American 
College’s  report.  Integrity  in  the  College  Curriculum  (1985),  criticized  the  quality  of 
American  higher  education  and  called  for  reform  and  assessment.  Educators,  like  Derek 
Bok  (1992),  former  president  of  Harvard,  and  Henry  Rosovsky  (1992),  former  dean  of  the 
faculty  of  arts  and  sciences  at  Harvard,  chastised  America’s  leading  institutions  for  not 
making  education  a  priority.  In  addition,  the  American  public  began  to  question  the  value 
of  higher  education  as  tuitions  continued  to  rise  even  as  state  and  federal  funds  available 
for  higher  education  decreased.  One  source  (Wallace,  1993)  claimed  that  in  the  1980s, 
state  and  local  government  expenditures  for  higher  education  were  placed  21st  in  a  list  of 
25  top  government  priorities  (p.  26).  As  available  funds  decreased  and  public  criticism 


increased,  it  became  paramount  that  colleges  and  universities  provide  empirical  evidence 
of  quality  and  effectiveness. 

To  answer  these  criticisms,  colleges  and  universities  created  assessment  programs  to 
meet  their  individual  institutional  needs  while  addressing  issues  of  accountability.  The 
external  push  to  create  assessment  programs  came  largely  from  state  governments.  In 
some  states,  direct  legislative  actions  forced  colleges  and  universities  to  adopt  assessment 
measures  on  their  campuses  (Terenzini,  1989).  While  some  institutions,  such  as  James 
Madison  University  and  the  University  of  Colorado  at  Boulder,  created  institution-wide 
assessment  programs  without  state  intervention  (Ewell,  1987),  the  assessment  movement 
was  largely  activated  by  external  sources  (Sims,  1992;  Terenzini,  1989).  While 
accountability-driven  assessment  responded  to  the  public’s  needs  and  concerns, 
institutions  found  that  mandated  assessment  reporting  could  also  be  beneficial  for  self¬ 
review,  renewal,  and  improvement  (Banta  and  Moffett,  1987;  Aper,  Cuver,  &  Hinkle, 
1990).  As  institutions  began  to  tie  their  efforts  to  the  teaching  and  learning  process  on 
their  campuses,  they  found  they  could  address  both  accountability  and  improvement 
(Banta,  1993). 

The  dichotomy  between  designing  programs  for  improvement  versus  accountability 
created  a  multitude  of  assessment  efforts  on  college  campuses.  A  thorough  and 
thoughtfixl  review  of  the  literature  revealed  that  there  were  numerous  examples  of 
assessment  programs  throughout  academe.  The  most  widely  discussed  and  used  method 
of  assessment  ties  excellence  in  education  to  student  outcomes  (El-Khawas,  1993). 
Student  outcomes  assessment,  as  developed  by  Alexander  Astin  (1993)  and  others 
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(Halpem,  1987;  Jacobi,  Astin,  and  Ayala,  1987)  describes  assessment  as  determining  the 
impact  of  college  on  students.  According  to  Astin  (1993),  “the  most  excellent  colleges 
and  universities  are  those  that  ‘add  the  most  value’  to  the  student’s  knowledge  and 
personal  development”  (p.  7).  While  student  outcomes  assessment  is  not  a  new  concept, 
the  current  movement  encompasses  the  whole  process  of  evaluating  educational  programs, 
including  establishing  goals,  gathering  data,  providing  feedback,  and  improvement 
measures  as  well  as  the  overall  issue  of  accountability  (Ewell,  1987).  The  “talent 
development”  or  “value-added”  model  of  assessment  uses  data  to  establish  causal 
relationships  between  the  college  environment  and  observed  student  outcomes  or  simply 
documents  students’  performance  at  a  single  point  in  time  (Jacobi,  Astin,  and  Ayala,  1987, 
p.  19). 

The  purpose  of  this  study  was  to  adapt  assessment  concepts  and  principles  that 
educators  have  used  successfully  in  higher  education  to  develop  a  comprehensive 
assessment  program  at  an  Air  Force  professional  military  education  institution.  Air 
Command  and  Staff  College  (ACSC).  ACSC  is  located  at  Maxwell  Air  Force  Base  in 
Montgomery,  Alabama.  It  is  a  ten-month  in-residence  program  designed  for  officers 
recently  promoted  to  the  rank  of  Major  (at  about  the  twelve-year  point  in  their  careers). 
For  Academic  Year  (AY)  1994-1995,  ACSC’s  curriculum  consisted  often  academic 
courses:  Campaign  Introduction  (20  hours).  Command  and  Leadership  (55  hours).  War 
Objectives  (26  hours).  Military  Theory  (48  hours).  Strategic  Structures  (87  hours). 
Operational  Structures  (90  hours).  Joint  Operations  and  Campaign  Concepts  (67  hours). 
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Air  Campaign  (136  hours),  War  Termination  (21  hours),  and  Campaign  2000+  (51  hours). 
(For  a  complete  description  of  these  courses,  please  see  appendix  B). 

Air  Force  professional  military  education  (PME)  spans  an  ofiScer’s  career  and  consists 
of  three  Colleges,  Squadron  OfBcers’  School  (SOS),  Air  Command  and  Staff  College 
(ACSC),  and  Air  War  College  (AWC).  As  the  mid-level  course,  ACSC  is  of  particular 
interest  because  it  serves  as  the  linchpin  in  the  overall  system.  ACSC  students  will  be  the 
backbone  of  the  officer  corps  for  decades  to  come.  Determining  how  best  to  assess  and 
improve  this  program  will  undoubtedly  lead  to  improvements  in  the  quality  and  strength  of 
future  Air  Force  leadership. 


Problem  Description 

Several  visits  to  ACSC  at  Maxwell  Air  Force  Base  in  Montgomery,  Alabama  coupled 
with  discussions  with  faculty,  students,  and  key  leaders  at  the  College  revealed  a  concern 
about  program  effectiveness  and  how  well  the  College  is  accomplishing  its  goals.  The 
program  evaluation  concerns  of  senior  leaders  revolved  around  two  major  issues:  How 
can  program  effectiveness  be  measured  and  how  can  the  results  of  such  measures  be  used 
to  change  and/or  improve  the  program? 

The  current  ACSC  evaluation  system  is  a  multi-tiered  method.  It  consists  of  an  end- 
of-  program  questionnaire  administered  to  students  before  their  departure  from  the  school, 
and  both  a  graduate  and  a  graduate’s  supervisor  surveys,  which  occur  three  years  after 
graduation.  ACSC  evaluators  compile  and  report  the  data  from  these  instruments  as 
statistical  descriptive  data,  consisting  largely  of  percentages.  They  highlight  negative 
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trends  and  distribute  the  reports  to  the  ACSC  Commandant’s  and  Dean  of  Faculty’s 
offices  and  to  anyone  else  who  wants  a  copy. 

While  the  evaluation  methods  currently  being  used  at  ACSC  provide  decision  makers 
with  some  information  about  the  overall  effectiveness  of  the  program,  they  consist  of 
largely  descriptive  and  anecdotal  evidence.  In  addition,  response  rates  tend  to  be  low. 
According  to  ACSC  evaluators,  the  response  rate  for  graduates’  surveys  in  the  past  was 
around  twenty  percent  (20  %).  This  low  rate  may  be  attributable  to  the  long  period  of 
time  between  graduates’  departure  from  the  school  and  when  they  received  the  surveys  (a 
period  of  three  years).  An  evaluation  plan  to  send  a  one-year  follow-up  survey  to 
graduates  and  their  supervisors  is  in  draft  format  but  it  has  yet  to  be  implemented.  There 
is  definitely  a  need  for  improved  instruments  and  survey  methodology  at  ACSC. 

This  dissertation  went  beyond  descriptive  evidence  by  also  providing  quantitative 
correlational  and  predictive  data.  Through  a  quantitative  methodology,  this  study 
examined  the  perceived  quality  of  ACSC’s  teaching  methods  and  program  activities.  It 
also  addressed  which  elements  of  the  program  contributed  most  to  students’  perceptions 
of  their  competence  on  outcome  variables.  This  dissertation  also  provided  evaluators  at 
ACSC  with  an  improved  survey  instrument  while  creating  a  statistical  data  base  which 
could  be  used  for  future  longitudinal  studies.  Finally,  by  pinpointing  which  elements  of  the 
program  contributed  most  to  students’  perceptions  of  competence  on  outcomes,  this  study 
will  lead  to  program  improvement.  By  giving  decision-makers  at  ACSC  specific 
information  about  the  program,  it  is  more  likely  that  they  will  use  the  results  to  bring 
about  changes. 
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These  issues  are  of  interest  to  all  stake  holders  at  ACSC.  Senior  Air  Force  leaders 
want  to  know  if  the  resources  expended  to  make  ACSC  a  first-rate  professional 
development  program  are  worth  the  cost  and  if  they  can  justify  future  expenditures. 
Program  administrators  and  developers  want  to  know  how  to  improve  the  course  fi-om 
year  to  year  while  instructors  seek  innovative  ways  to  fecilitate  learning  in  the  classroom. 
Students  who  attend  ACSC  hope  that  the  value  of  the  experience  is  worth  the  time 
expended  and  how  it  can  help  them  be  better  Air  Force  officers. 

In  summary,  current  assessments  of  ACSC  may  lead  to  program  improvement,  but 
because  the  evaluation  methods  are  largely  subjective  and  descriptive,  the  information  is 
incomplete.  This  study  will  provide  key  stake  holders  at  ACSC  with  objective 
correlational  and  predictive  data  that  will  indicate  connections  between  program  elements. 


Research  Questions 

The  major  question  this  study  addressed  was  an  evaluation  question. 

How  effective  was  ACSC  in  meeting  its  goals  and  what  contributed  to  this 
effectiveness  as  perceived  by  the  students  who  graduated  from  the  program? 

This  question  revolved  around  the  overall  effectiveness  of  ACSC.  Effectiveness  was  the 

extent  to  which  students  rated  their  competencies  on  outcome  variables.  The  question 

also  involved  determining  if  students’  perceptions  of  the  quality  of  environmental  variables 

affected  their  perceived  competencies  on  outcome  goals. 
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To  answer  the  above  question,  the  researcher  divided  the  subsidiary  questions  into 
four  categories;  descriptive,  relational,  predictive,  and  implications.  First,  the  researcher 
statistically  described  all  three  categories  of  variables.  Specific  descriptive  questions  were: 


1.  What  were  the  demographic  factors  and  student  expectations  (inputs)  of  ACSC 
graduates  in  academic  year  (AY)  1994-1995? 

2.  How  did  students  rate  the  quality  of  the  teaching  methods  and  program  activities 
(environmental  factors)  at  Air  Command  and  Staff  College? 

3.  How  did  students  rate  their  competency  on  outcome  measures  at  the  time  they 
completed  Air  Command  and  Staff  College? 

Second,  the  researcher  wanted  to  know  the  relationships  between  the  independent  and 
dependent  variables.  Specific  correlational  questions  were: 

4.  Was  there  a  relationship  between  input  variables  (demographics  aud  student 
expectations)  and  student  outcomes? 

5.  Was  there  a  relationship  between  environmental  variables  (teaching  methods  and 
program  activities)  and  student  outcomes? 

Third,  the  researcher  wanted  to  make  predictive/causal  claims  about  the  relationship 
between  environmental  factors  and  outcomes.  To  ensure  that  claims  made  about  the 
effect  of  certain  environmental  factors  on  outcomes  were  accurate,  the  researcher 
“controlled”  for  other  variables  that  might  affect  outcomes.  Since  input  variables  may 
affect  the  relationship  between  environmental  variables  and  outcomes,  the  next  question 
was; 

6.  Holding  the  input  variables  (demographics  and  student  expectations)  constant, 
was  there  a  relationship  between  environmental  variables  (teaching  methods  and 
program  activities)  and  student  outcomes? 


8 


Finally,  based  on  the  quantitative  data  compiled  from  the  questions  above  and  from  open- 
ended  questions  posed  to  academic  year  1994-1995  graduates,  the  researcher  made 
recommendations  about  program  changes  and/or  improvements.  The  final  question 
addressed  this  issue: 

7.  What  were  the  implications  of  the  results  of  this  study  for  Air  Command  and 
Staff  College? 

Conceptual  Framework 

This  dissertation  adapted  a  model  created  and  implemented  by  Alexander  Astin  (1993, 
first  published  in  1991),  the  director  of  the  Higher  Education  Research  Institute  of  the 
University  of  California  at  Los  Angeles.  Astin  refers  to  his  model  of  assessment  as  the 
input-environment-outcome  (I-E-0)  model  which  he  has  used  largely  to  assess  students  in 
college  and  university  settings.  According  to  Astin,  program  evaluation  and  assessment 
can  be  used  to  inform  decision-makers  about  the  overall  effectiveness  of  their  program  if  it 
focuses  on  student  talent  development  (outcomes).  Student  talent  development  refers  to 
the  extent  students  change  as  a  result  of  their  educational  experience.  Talent  development 
can  consist  of  cognitive  and/or  affective  changes.  Such  a  model  links  student  outcomes  to 
inputs  and  to  environmental  factors  occurring  in  the  learning  environment. 

Educational  assessment,  according  to  Astin,  must  include  “data  on  student  inputs, 
student  outcomes,  and  the  educational  environment  to  which  the  student  is  exposed”  (p. 
18).  Within  the  I-E-0  model,  inputs  (I)  refer  to  the  qualities  that  students  bring  with  them 
as  they  enter  the  educational  environment.  The  environment  (the  E  in  the  model)  consists 
of  everything  that  happens  to  a  student  while  he  or  she  is  part  of  the  program.  Outcomes 
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(O)  are  the  talents  the  educational  program  will  develop  in  its  students.  Most  educators 
look  at  the  relationship  between  “E”  and  “O,”  the  effects  of  the  environmental  variables  on 
outcome  variables.  The  researcher  who  uses  Astin’s  model,  however,  also  looks  at  the 
relationship  between  “I”  and  “O.”  Represented  graphically,  the  I-E-0  model  looks  like 
this: 


In  this  study,  the  input  and  environmental  variables  were  the  independent  variables  and  the 
outcome  variables  were  the  dependent  variables. 

Overall  Design 

This  study  was  a  cross-sectional  study  designed  to  evaluate  an  Air  Force  educational 
program,  ACSC,  based  on  the  perceptions  of  graduates  of  the  program  in  AY  1994-1995. 
The  researcher  devised  a  90-item  questionnaire  titled,  “Student  Perceptions  of  Program 
Effectiveness  Questionnaire”  (Appendbc  A),  to  collect  data  on  the  three  categories  of 
variables--  inputs,  environment,  and  outcomes.  These  variables  are  defined  as  followed: 

Input  Variables 

Input  variables  were  those  characteristics  that  students  brought  with  them  to  the 
program.  ACSC  students,  unlike  traditional  college  students,  come  fi'om  a  homogeneous 
population.  They  enter  the  program  at  roughly  the  same  point  in  their  careers  and  bring 


with  them  many  of  the  same  experiences.  In  choosing  input  variables  for  this  study,  the 
researcher  was  concerned  with  those  characteristics  on  which  the  students  were  more 
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likely  to  vary.  Input  variables  were  also  chosen  based  on  their  potential  for  having  an 
effect  on  student  perceptions  of  their  competence  on  outcomes.  This  study  focused  on 
two  categories  of  student  input  characteristics:  demographics  (age,  gender,  race,  career 
field.  Master  degree  concentration,  and  commission  source)  and  student  expectations. 
Student  expectations  consisted  of  nine  items  on  which  students  were  asked  to  choose  all 
that  apply  (see  question  88  on  the  questionnaire.  Appendix  A). 

Environmental  Variables 

Environmental  variables  were  the  program  elements  and  teaching  methods  that 
students  experienced  while  at  ACSC.  The  environmental  variables  fell  into  sbc  categories: 
Teaching  Methods,  Technology,  Curriculum,  Research,  Faculty,  and  Grading.  These 
variables  were  derived  through  discussions  with  principle  stake  holders  at  ACSC  about 
what  variables  they  deemed  important  to  be  included  in  this  evaluation. 

Outcome  Variables 

Outcome  variables  were  the  talents  that  the  program  claimed  to  instill  in  its  students. 
Talents,  in  this  case,  were  defined  as  the  course  goals  outlined  by  the  faculty  and  staff  at 
ACSC  in  their  course  syllabi.  By  distilling  information  from  the  nine  course  syllabi,  the 
researcher  identified  four  major  divisions  of  outcome  variables:  Joint  Campaign  Planning 
Process,  Air/Space  Power  Applications,  Command  and  Leadership,  and  Critical  Thinking. 
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The  Questionnaire 

The  questionnaire  consisted  of  six  sections  corresponding  to  the  three  categories  of 
variables  in  the  model  (inputs,  environment,  or  outcomes)  except  section  “A”  that 
consisted  of  four  questions  required  by  Air  University.  These  four  questions  were 
“quality”  questions  referring  to  Total  Quality  Management  principles  adopted  by  the  Air 
Force  and  results  from  this  section  were  not  analyzed  or  reported  m  this  study.  Input 
variables,  consisting  of  demographics  and  student  expectations,  were  measured  by 
students’  responses  on  Section  “D”  and  “E”  (questions  82-88),  Environmental  variables 
corresponded  to  Section  “C,”  “Program  Activities  and  Teaching  Methods”  (items  50-81). 
Graduates  were  asked  to  rate  the  quality  of  program  elements  used  at  ACSC  on  a  five- 
point  scale  ranging  from  “unsatisfrictory”  to  “outstanding.”  Outcome  variables  were 
measured  by  students’  responses  to  the  question:  “How  would  you  rate  your  competency 
on  the  following  items  at  the  point  in  time  when  you  completed  Air  Command  and  Staff 
College”  (items  5-49)?  Items  were  rated  on  a  five-point  scale,  ranging  from  “no 
competence”  to  “great  competence.” 

The  initial  questionnaire  packets  were  mailed  on  November  3,  1995  to  all  395  Air 
Force  officer  graduates  of  AY  1994-1995.  This  date  for  the  initial  mailing  served  two 
purposes.  It  allowed  time  for  the  researcher  to  conduct  a  pilot  study  of  the  instrument 
which  was  accomplished  during  the  week  of  August  14-18, 1995.  (For  a  detaOed 
description  of  the  pilot  study,  please  see  Chapter  3,  “Methodology”).  In  addition  to 
providing  time  for  a  pilot  study,  by  waiting  until  November  to  mail  out  the  questionnaires, 
the  researcher  hoped  that  graduates  would  have  a  chance  to  complete  their  moves  from 
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ACSC  and  settle  into  their  jobs.  Most  ACSC  graduates  go  to  new  positions  upon 
graduation  from  the  College  and  it  takes  several  months  for  them  to  make  the  transition. 

Study  Group 

The  AY  1994-1995  student  body  consisted  of  approximately  580  students,  but  this  figure 
included  international  students,  US  civilians,  and  individuals  from  the  other  services.  This 
study  was  only  concerned  with  the  perceptions  of  United  States  Air  Force  oflBcers  which 
totaled  395  students.  Mailing  labels  were  obtained  from  the  Air  Force  Personnel  Center 
and  questionnaires  were  mailed  to  all  395  Air  Force  oflBcer  graduates  from  the  AY  1994- 
1995  Class.  A  total  of 228  usable  questionnaires  were  returned  for  a  return  rate  of  fifty- 
eight  percent  (58  %).  However,  out  of  the  original  395  questionnaires  mailed,  107  were 
returned  “undeliverable  as  addressed,”  bringing  the  total  number  of  questionnaires  to 
reach  participants  to  288.  Based  on  this  figure,  the  return  rate  was  seventy-eight  percent 
(78  %). 

Data  Analyses 

After  receiving  and  scoring  the  questionnaires,  the  researcher  approached  data 
interpretation  in  four  steps.  First,  descriptive  statistics  such  as  the  mean,  standard 
deviation  and  percentages  characterized  each  category  of  variable.  These  statistics 
answered  questions  1-3.  Second,  the  researcher  used  correlational  statistics  to  determine 
the  relationships  between  the  independent  (inputs  and  environment)  and  dependent 
(outcomes)  variables.  To  look  at  the  relationship  between  demographic 
characteristics/student  expectations  (inputs)  and  outcome  variables,  the  researcher  used 
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the  cross  tabulation  method.  This  method  summarized  the  degree  of  relationship  between 
inputs  and  outcomes  and  data  were  reported  as  percentages.  The  Pearson  product- 
moment  correlation  coefficient  (Pearson  “r”)  was  used  to  determine  to  what  extent 
environmental  variables  correlated  with  outcome  variables.  These  statistical  methods 
answered  questions  4  and  5.  Third,  along  with  descriptive  correlational  data,  the 
researcher  used  multiple  regression  analysis  to  predict  which  environmental  variables  most 
affected  perceived  student  competency  on  outcome  variables  after  the  effects  of  input 
variables  were  controlled.  By  holding  constant  the  input  variables,  the  researcher  was  able 
to  identify  the  environmental  factors  that  most  affected  student  perceived  competency  on 
any  given  outcome  measure.  This  method  answered  question  6.  Finally,  the  answer  to 
question  7  came  from  an  overall  analysis  of  the  data  based  on  results  from  the  above 
methods  and  open-ended  items  from  the  questionnaire.  This  question  involved  the 
researcher  making  recommendations  about  change  to  or  improvements  in  the  program. 
This  was  the  least  objective  part  of  the  study.  Although  the  researcher  was  subjective 
when  making  recommendations,  the  objective  data  presented  in  this  study  suggested  the 
necessary  steps  that  stake  holders  may  take.  In  addition  to  the  quantitative  data,  the 
researcher  also  reported  results  of  the  open-ended  questions  on  the  questionnaire 
(questions  89  and  90).  These  data  were  analyzed  and  placed  in  “Uke”  categories  to 
determine  patterns.  The  results  of  this  content  analysis  were  reported  in  a  separate  section 
after  the  quantitative  data. 
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Expected  Outcomes  and  Limitations 

ACSC  made  significant  changes  to  its  curriculum  in  the  last  two  years.  They  attempted 
to  alter  their  philosophical  approach  to  education  while  upgrading  their  image  as  an 
educational  institution.  The  “revolution”  at  the  College  created  much  discussion,  not  to 
mention  controversy,  about  the  role  of  professional  military  education  (PME)  in  the 
modem  military.  Innovations  in  technology,  research,  and  teaching  methodology  brought 
ACSC  accolades  fi’om  senior  Air  Force  leaders  and  civilian  educators  alike.  It  also 
brought  money  and  prestige  to  the  College  as  a  military  institution.  However,  most  of  the 
existing  evidence  about  the  success  of  ACSC  is  anecdotal  and  descriptive  and  has  yet  to 
make  its  way  into  the  mainstream  of  higher  education  literature.  By  tying  this  study  to 
important  issues  in  higher  education,  the  researcher  brought  a  research  element  to  ACSC’s 
primary  stake  holders  of  which  they  were  unaware. 

This  dissertation  produced  three  major  outcomes.  It  provided  objective  quantitative 
data  that  supplemented  existing  anecdotal,  descriptive  results  from  previous  evaluations. 
Quantitative  evidence  gave  key  Air  Force  leaders  and  stake  holders  at  ACSC  additional 
information  they  will  need  to  make  judgments  about  the  value  of  their  programs  and 
professional  military  education  as  a  whole.  While  supplying  objective  data  to  supplement 
existing  research,  this  study  also  built  a  foxmdation  on  which  to  base  additional  research. 
The  researcher  believes  that  this  work  will  create  discussions,  debates,  and  shared 
scholarship  about  the  uses  and  limitations  of  professional  military  education.  Presently, 
the  literature  lacks  any  significant  discourse  about  military  education  scholarship.  Finally, 
this  dissertation  brought  significant  higher  education  issues  and  research  methodology  to 
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the  attention  of  military  educators.  Assessment  issues  and  accountability,  for  example,  are 
presently  important  topics  in  the  higher  education  literature.  Educators  at  ACSC  should 
concern  themselves  with  the  same  issues  and  problems  that  are  occurring  throughout 
academia.  However,  military  organizations  tend  to  rely  on  the  professional  opinions  of  a 
few  when  facilitating  change.  This  study  illuminated  some  important  educational  theories 
and  practices  that  will  aid  military  educators  at  ACSC  in  creating  an  educational 
environment  based  on  seminal  issues  in  the  field. 

While  the  researcher  expected  this  dissertation  to  provide  ACSC  with  quantitative 
objective  data  that  will  lead  to  program  improvement,  there  were  limitations  to  the  study 
as  well.  First,  the  research  was  limited  by  time  and  proximity  to  the  College  which  made  a 
longitudinal  study  impractical.  While  student  outcomes  assessment  works  best  when  the 
researcher  can  rely  on  longitudinal  data  based  on  the  results  spanning  several  years,  it  was 
not  possible  for  this  study  to  implement  such  a  plan.  The  study  also  may  be  hindered  by  a 
design  decision  relating  to  the  scope  of  the  study.  This  dissertation  was  designed  to 
provide  a  comprehensive  evaluation  of  ACSC.  As  a  result,  every  element  of  the  program 
could  not  be  investigated  in  depth.  The  researcher  had  to  make  decisions  about  what 
elements  should  and  should  not  be  included  in  the  evaluation.  Finally,  this  study  utilized 
students’  perceptions  rather  than  measuring  outcomes  directly  through  a  pretest/posttest 
design.  However,  future  studies  should  consider  a  longitudinal  design  based  on  pretest  as 
well  as  posttest  data. 

These  limitations  notwithstanding,  the  researcher  believes  that  the  data  gathered  in  this 
study  will  provide  decision  makers  at  ACSC  with  a  valuable  and  timely  assessment  of  their 
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program.  While  student  perceptions  of  program  effectiveness  is  only  one  way  in  which  to 
measure  outcomes,  it  offers  an  important  perspective  about  what  elements  of  ACSC  work 
best  in  facilitating  high  competence  on  outcome  variables,  leading  to  program 


improvement. 
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CHAPTER  TWO 

LITERATURE  REVIEW 

In  the  higher  education  literature  the  terms  “assessment  and  “evaluation”  are  often 
used  interchangeably.  This  may  cause  confusion,  but  it  is  often  hard  to  draw  a  distinction 
between  the  two  terms  (Sims,  1992).  A  review  of  the  literature  reveals  that  there  is  no 
consensus  on  what  the  terms  “assessment,”  or  “evaluation”  mean  or  what  processes  they 
comprise.  The  term  “assessment”  first  emerged  in  the  late  1930s  when  Henry  Mvuray 
used  it  to  refer  to  the  appraisal  of  individuals.  Conversely,  educators  often  use  the  term, 
“evaluation,”  to  define  program  and  group  assessments  (Madaus,  Stuffiebeam,  &  Scriven, 
1983;  Sims,  1992).  Some  writers  (Boyer  and  Ewell,  1988;  Lenning,  1980)  make  the 
distinction  between  assessment  and  evaluation  by  defining  “assessment”  as  what  occurs 
when  measurements  are  analyzed,  while  “evaluation”  occurs  when  judgments  are  made 
about  assessment  results.  While  others  (Conrad  and  Eagan,  1989;  Gardner,  1990)  use  the 
words  to  mean  the  same  thing.  For  the  purposes  of  this  study  the  two  terms  of 
“assessment”  and  “evaluation”  will  mean  “a  common  efibrt  to  imderstand  and  judge  the 
merit  or  worth  of  teaching  and  learning  within  a  course,  curriculum,  educational  program, 
sequence  of  study,  department,  unit,  or  institution”  (Davis,  1989,  p.  8).  Since  the  overall 
goal  of  this  dissertation  was  to  determine  the  effectiveness  (the  merit  or  worth)  of  ACSC 
and  what  contributed  to  that  effectiveness  as  perceived  by  graduates  of  the  program,  this 
broad  view  of  assessment  and  evaluation  was  appropriate. 
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This  study  was  grounded  in  the  literature  that  emerged  out  of  the  assessment 
movement  which  began  in  the  mid  1980s.  This  literature  review  will  begin  by  focusing  on 
the  comparisons  between  the  assessment  and  evaluation  movements.  It  will  then  discuss 
state  level  mandates  and  some  important  “impact  studies.”  Next,  the  discussion  will 
highlight  some  institutional  assessment  programs,  and  finally,  it  will  review  ACSC 
evaluation  programs  and  policies. 

The  Evaluation  and  Assessment  Movements  Compared 

While  there  does  not  appear  to  be  a  consensus  about  what  one  means  when  referring  to 
“evaluation”  or  “assessment,”  the  two  terms  represent  separate  movements  with  distinct 
methodologies,  processes,  and  literatures.  The  roots  of  educational  evaluation  can  be 
traced  back  to  the  early  nineteenth  century  and  the  Industrial  Revolution.  This  period  in 
American  society  was  marked  by  attempts  to  reform  educational  and  social  programs. 

The  first  processes  designed  to  evaluate  performance  of  schools  in  the  United  States 
occurred  in  Boston  in  1 845  with  the  use  of  student  test  scores  to  evaluate  school 
effectiveness.  A  few  years  later,  Samuel  Gridley  Howe  and  Horace  Mann  advocated 
written  essays  as  evaluation  tools.  Between  1887  and  1898,  Joseph  Rice  conducted  what 
is  viewed  as  the  first  formal  educational  evaluation  in  America  by  using  test  scores  to 
evaluate  spelling  instruction.  By  the  turn  of  the  century,  femous  researchers  such  as  Ayers 
and  Thorndike  were  using  standardized  tests  to  yield  normative  data  which  enabled  one 
school  system  to  compare  itself  with  another  (Madaus,  Stufflebeam,  &  Scriven,  1983). 
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By  the  1930s,  the  focus  of  school  evaluation  changed  with  the  work  of  Ralph  W. 

Tyler.  Tyler  coined  the  term  “educational  evaluation”  and  his  methods  centered  on  the 
extent  to  which  objectives  were  achieved  as  part  of  an  instructional  program.  Tyler 
approached  evaluation  by  comparing  intended  objectives  with  actual  outcomes.  As  a 
result  of  his  work  in  the  1930s  and  1940s,  evaluators  and  educators  began  to  view 
evaluation  in  a  much  broader  sense,  covering  a  wider  range  of  outcome  variables  than 
those  measured  by  standardized  tests  (Madaus,  Stufflebeam,  &  Scriven,  1983). 

Although  Tyler’s  methods  brought  innovation  to  educational  evaluation  practices,  the 
use  of  standardized  tests  did  not  disappear.  In  fact,  in  the  1950s  and  1960s,  the  use  of 
standardized  tests  increased  as  new  technological  advances  allowed  researchers  to 
machine  score  tests.  Many  new  nationally  standardized  tests  were  developed  during  this 
period  and  schools  purchased  them  in  large  numbers.  As  a  result  of  the  increased  use  of 
standardized  tests  to  measure  effectiveness,  professional  organizations  imposed  steps  to 
regulate  test  taking  activities.  In  addition,  the  principles  of  experimental  design  and 
statistical  analysis  would  become  the  most  popular  methods  used  in  education  evaluation. 
Evaluators  during  this  period  also  attempted  to  regulate  Tylerian  methods  by  developing 
taxonomies,  such  as  the  taxonomy  created  by  Bloom,  in  order  to  help  guide  which 
objectives  were  worthy  of  evaluation  (Madaus,  Stufflebeam,  &  Scriven,  1983). 

Educational  program  evaluation  in  the  1960s  looked  much  like  it  did  in  the  early  part 
of  the  century,  relying  on  standardized  tests  and  quantitative  data  collection  techniques. 
What  had  changed  by  the  mid-1960s,  however,  was  the  scope  of  evaluations.  With  the 
Soviet  launching  of  Sputnik  in  1957  and  the  subsequent  federally  enacted  National 
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Defense  Education  Act  of  1958,  large  scale  curriculum  evaluation  efforts  were  underway. 
These  evaluations  were  designed  to  look  at  the  curriculum  of  the  new  educational 
programs  in  science,  mathematics,  and  foreign  language  that  the  National  Defense  Act 
created.  In  addition  to  the  National  Defense  Act  of  1 958,  further  federal  legislation  in  the 
1960s  spurred  on  by  the  efforts  of  Senator  Hubert  Humphrey  and  Presidents  Kennedy  and 
Johnson,  led  to  more  large-scale  evaluative  efforts  (Madaus,  Stufflebeam,  &  Scriven, 
1983).  The  educational  evaluators  of  the  1960s  were  responding  to  a  national  need  to 
evaluate  new  curricula.  At  first  they  were  not  so  much  concerned  about  the  relevance  and 
utility  of  their  findings,  but  were  reacting  to  federal  pressure  to  provide  information, 
regardless  of  its  quality  or  meaning.  Even  those  who  were  technically  competent  in 
quantitative  research  methods  found  the  task  of  evaluating  their  programs  difficult. 
Meeting  local  objectives  while,  at  the  same  time,  providing  Congress  with  information 
about  the  extent  to  which  their  programs  were  accomplishing  national  objectives  proved 
to  be  a  formidable  task.  (Worthen  and  Sanders,  1991).  It  soon  became  evident  that  new 
approaches  to  educational  evaluation  were  needed.  Cronbach  (1963)  was  perhaps  one  of 
the  first  to  argue  that  educational  evaluation  should  take  a  new  direction.  He  claimed  that 
comparisons  between  programs  were  not  as  important  as  gathering  and  reporting  data  that 
would  help  program  managers  develop  new  curriculum.  Scriven  (1967)  followed  with  a 
paper  on  the  methodology  of  evaluation.  Cuba  (1967)  voiced  dissatisfaction  with  the 
traditional  quantitative  methods  of  evaluating  educational  programs.  Articles  by  Stake 
(1967)  and  Stufflebeam  (1968)  also  added  to  the  emerging  literature  on  educational 
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During  the  1970s  and  continuing  to  the  present  time,  the  field  of  evaluation  evolved 
into  a  distinct  profession.  A  number  of  journals  such  as  Educational  Evaluation  and 
Policy  Analysis,  Evaluation  Review,  and  New  Directions  for  Program  Evaluation,  to 
name  but  a  few,  were  begun.  There  were  numerous  books  and  monographs  dealing  with 
evaluation  and  many  universities  began  to  offer  courses  in  evaluation  methodology. 

Several  universities  now  offer  graduate  programs  in  evaluation  and  professional 
organizations  like  the  American  Evaluation  Association  provide  vehicles  for 
communicating  information.  (Madaus,  Stufflebeam,  &  Scriven,  1983).  As  the  literature 
expanded  and  the  discussions  became  more  open,  the  practice  and  theories  of  evaluation 
became  more  divergent.  Models  of  evaluation  proliferated  as  more  and  more  evaluators 
attempted  to  develop  new  and  better  ways  to  define  and  describe  the  nature  of  educational 
evaluation.  Worthen  and  Sanders  (1987)  explained  why  there  are  so  many  different 
approaches  to  evaluation  when  they  wrote,  “(l)ike  so  many  other  young,  emerging  fields, 
evaluation  is  troubled  by  definitional  and  ideological  disputes.  Those  who  write  about 
evaluation  differ  widely  in  their  views  of  what  evaluation  is  and  how  one  should  go  about 
doing  it”  (p.  43).  Worthen  and  Sanders  claim  that  there  have  been  over  50  different 
evaluation  models  developed  in  the  past  twenty  years. 

Although  evaluators  have  become  more  professionalized  and  numerous  models 
have  emerged,  there  is  still  little  consensus  about  evaluation  methods  and  techniques.  The 
continuing  debate  between  quantitative  and  qualitative  approaches  is  one  example  of  the 
kind  of  polarization  that  can  be  found  among  evaluators.  In  spite  of  the  sustained  debates, 
educational  evaluation  as  a  field  sufiers  from  the  lack  of  innovation.  According  to 
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Madaus,  StufHebeam,  and  Scriven  (1983),  “there  is  a  need  for  expanded  efforts  to  educate 
evaluators  to  the  availability  of  new  techniques,  to  try  out  and  report  results  of  using  the 
new  techniques,  and  to  develop  additional  techniques”  (p.  18).  Currently,  popular 
approaches  to  program  evaluation  do  not  center  around  a  single  dimension.  Rather, 
program  evaluations  can  take  many  forms.  Michael  Scriven  writes  (1993)  that  “different 
evaluation  designs  are  usually  required  for  ranking,  grading,  scoring,  and  apportioning”  (p. 
68).  Although  methodological  debates  dominate  the  evaluation  literature,  there  is  an 
increasing  concern  about  the  utilization  of  evaluation  results.  Evaluators  like  Ernest 
House  (1972),  Michael  Patton  (1986),  and  Carol  Weiss  (1972)  write  that  utilization  is 
one  of  the  central  problems  facing  evaluators  today.  Undoubtedly,  with  increased 
demands  for  accoimtability  and  continued  government  control  of  educational  evaluations, 
discussions  about  what  is  being  done  with  evaluation  results  will  necessarily  frame  the 
debate  well  into  the  twenty-first  century. 

Like  the  literature  on  educational  evaluation,  the  assessment  literature  deals  largely 
with  the  issue  of  accountability  versus  utility  of  results.  However,  unlike  educational 
evaluation,  those  involved  in  assessment  efforts  in  higher  education  have  not  become 
professionalized,  although  much  has  been  written  about  the  “assessment  movement.” 
Because  the  assessment  movement  has  recently  taken  a  front  seat  in  American  higher 
education,  it  may  appear  as  if  it  is  a  new  concept.  In  reality,  student  assessment  initiatives 
and  the  debates  over  accountability  have  always  been  a  part  of  the  history  of  American 
higher  education.  As  already  stated,  the  term  “assessment”  began  appearing  in  the  late 
1930s  and  has  often  been  used  to  refer  to  the  appraisal  of  individuals  (Sims,  1992).  A 
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great  deal  of  the  early  literature  on  assessment  revolves  around  the  use  of  achievement 
tests.  However,  the  early  use  of  the  word  “assessment”  to  mean  measurement  was 
radically  altered  in  the  mid  1 980s  as  the  accountability  crisis  gained  momentum.  The  call 
for  increased  accoimtability,  coupled  with  externally  mandated  assessment  initiatives 
during  that  period,  brought  about  more  comprehensive  assessment  measures.  These  new 
measures  encompassed  the  whole  process  by  which  one  evaluates  educational  programs  in 
higher  education,  including  establishing  goals,  gathering  data,  providing  feedback,  and 
improvement  measures  as  well  as  the  overall  issue  of  accountability  (Ewell,  1987). 

The  Assessment  Movement  and  Accountability 

In  1 974,  Howard  Bowen  commented  that  “[t]he  call  for  accountability  has  been  heard 
throughout  the  land”  (p.  xi).  Accountability,  he  says,  refers  to  the  obligation  that 
institutions  and  their  members  have  to  produce  outcomes  that  are  consistent  with  their 
goals.  Bowen  made  these  comments  a  decade  before  the  assessment  movement  began  in 
earnest,  but  even  then,  he  called  for  improvements  in  the  evaluation  of  performance  in 
higher  education.  “The  benefits,”  he  said,  “lie  not  only  in  satisfying  the  demands  of 
interested  outsiders. . .  [but]  also  extend  to  improving  the  effectiveness  and  efficiency  of 
internal  operations”  (p.  119-120). 

A  decade  after  Bowen  made  these  statements,  the  Southern  Regional  Education  Board 
(1984)  concluded  that  accountability  should  be  based  on  the  “demonstrated  achievement 
of  students,  not  just  on  financial  criteria”  (p.  42).  According  to  one  writer  (Folger,  1984), 
this  conclusion  was  representative  of  the  concern  of  legislators  and  the  public  that 
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education  should  do  a  better  job  and  be  able  to  show  that  they  are  doing  a  better  job  (p. 
75).  This  report  largely  directed  its  criticisms  toward  public  education  but  the  movement 
quickly  spread  to  higher  education.  The  pressure  to  present  evaluative  evidence  came 
from  inside  as  well  as  outside  academia.  Reports  such  as  Involvement  in  Learning  (Study 
Group  on  the  Conditions  of  Excellence  in  American  Higher  Education,  1984)  and  William 
J.  Bennett’s  monograph,  To  Reclaim  a  Legacy:  A  Report  on  the  Humanities  in  Higher 
Education  (1984)  dealt  with  the  quality  of  American  higher  education.  In  addition,  the 
Association  of  American  College’s  report,  Integrity  in  the  College  Curriculum,  called  for 
assessment  procedures  as  well  as  curricular  reform.  In  1986,  the  National  Governor’s 
Association  (NGA)  formed  a  Task  Force  on  College  Quality  to  look  at  how  colleges  and 
universities  could  demonstrate  student  learning  outcomes.  The  NGA  made  its  first  report 
in  1987,  indicating  that  thirteen  states  had  formal  college  quality  assessment  programs.  By 
1989,  according  to  the  NGA,  that  number  had  increased  to  twenty-eight  (National 
Governors  Association,  1989,  p.  42). 

While  educators  and  legislators  began  to  look  seriously  at  the  quality  of  American 
educational  programs,  the  American  public  brought  another  dimension  to  the  argument. 
Faced  with  media  reports  about  students  who  graduate  but  could  not  read  and  professors 
who  do  not  teach,  public  opinion  of  higher  education  turned  sour.  They  began  to  question 
not  only  “What  am  I  getting  for  my  money?”  but  “What  are  students  learning?”  Educators 
found  themselves  in  the  position  of  defending  what  they  do  to  the  public  at  large  (Erwin, 


1991,  p.  5). 
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The  assessment  movement  has  become  synonymous  with  the  accountability  movement, 
calling  for  higher  education  to  examine  itself  and  to  report  the  findings  to  its  many 
constituencies  (V.  McMillan,  1994).  This  “New  Accountability”  (Ewell  and  Jones,  1985) 
expects  colleges  to  develop  comprehensive  student  outcomes  assessment  programs  and  to 
implement  change  on  their  campuses.  By  1993,  97  percent  of  institutions  had  some  type 
of  assessment  activity  and  43  percent  described  their  programs  as  “extensive”  (El-Khawas, 
1993).  This  evidence  indicates  that  the  assessment  movement  is  here  to  stay  and  will 
continue  to  facilitate  change  on  college  campuses  across  the  country. 

Externally  Mandated  Assessment 

Although  the  idea  of  accountability  and  assessment  are  not  new  to  higher  education, 
what  makes  the  current  assessment  movement  significant  is  that  it  largely  has  been 
activated  by  external  sources  (Sims,  1992;  Terenzini,  1989).  While  the  federal 
government  and  regional  accrediting  agencies  have  indirectly  pressed  colleges  and 
universities  to  create  assessment  programs,  the  largest  impetus  for  change  has  come  from 
state  authorities.  In  some  states,  governors,  such  as  Thomas  Kean  of  New  Jersey,  John 
Ashcroft  of  Missouri,  and  Lamar  Alexander  of  Tennessee,  were  influential  in  carrying  the 
assessment  movement  forward.  (Aper,  Cuver,  &  Hinkle,  1990).  In  some  states,  such  as 
Colorado  (House  Bill  1187),  Virginia  (Resolution  196)  and  California  (Concurrent 
Resolution  141),  direct  legislative  action  spurred  assessment  measures  and  institutional 
reform.  Other  state  actions  originated  at  the  coordinating  or  governing  board  level 
independent  of  explicit  legislation,  such  as  in  the  case  of  New  Jersey  or  Maryland  (Ewell, 
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1991).  This  external  push  is  significant  because  it  forced  colleges  and  universities  to  create 
systematic,  campus-wide  assessments  that  encompassed  a  wide  range  of  activities 
(Terenzini,  1989). 

Some  colleges  and  universities  created  institution-wide  assessment  procedures  without 
direct  state  intervention.  In  these  institutions,  it  was  the  feculty  or  administrative 
committees  who  created  assessment  programs,  procedures  and  policies.  For  example,  at 
James  Madison  University  (JMU),  four  different  assessment  models  were  tried  in  the 
departments  of  nursing,  economics,  Russian  language,  and  theatre.  Those  responsible  for 
assessment  at  the  school  compared  experiences  fi’om  each  department  to  learn  the 
strengths  and  weaknesses  of  various  models  of  assessment.  Similarly,  at  the  University  of 
Colorado,  Boulder,  individual  departments  developed  their  own  assessment  programs, 
centering  on  disciplinary  concerns  (Ewell,  1987). 

Just  as  states  vary  in  their  level  of  state  mandated  assessment,  they  also  vary  widely  on 
their  assessment  policies.  Some  states  mandate  statewide  standardized  testing  while 
others  require  institutional  reporting  of  different  indicators  of  effectiveness.  South 
Dakota,  for  example,  requires  that  all  public  institutions  test  students  for  their 
demonstrated  performance  in  their  major  as  a  criterion  for  successful  completion  (Ewell, 
1985).  In  Missouri,  legislators  instituted  state  mandated  sophomore  testing  at  the 
University  of  Missouri,  Columbia,  to  attend  to  perceived  problems  at  that  campus  alone. 
The  mandated  testing,  in  this  case,  was  institution-specific,  leaving  the  other  campuses  of 
the  University  system  alone  (Ewell,  1991).  Private  colleges  have  not  been  immune  fi’om 
state  mandated  testing.  For  example,  at  Xavier  University  in  Louisiana,  the  state  requires 
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that  the  college  administer  a  “rising  junior”  exam  (Ewell,  1987).  While  some  states  call 
for  mass  testing  as  evidence  of  effectiveness,  the  most  common  procedure  is  for  states  to 
set  guidelines  for  institutional  reporting  that  provides  a  plan  and  an  assessment  of 
effectiveness  measures.  South  Carolina  requires  each  university  to  submit  a  report 
covering  eighteen  categories  of  outcome  data.  In  Virginia,  the  state  mandated  assessment 
guidelines  require  that  institutions  report  quantifiable  data  on  four  outcome  measures: 
outcomes  in  the  major  and  general  education;  basic  skills  proficiency;  remediation  in 
verbal  and  quantitative  skills;  and  alumni  follow-up  (Sims,  1992). 

Institutions  have  largely  responded  to  externally  mandated  assessment  by  designing  and 
implementing  assessment  procedures  geared  toward  accountability.  Because  states 
demand  that  colleges  produce  evidence  that  they  are  accomplishing  what  they  say  they 
are,  it  makes  sense  that  institutions  tie  their  assessment  measures  to  the  accountability 
movement.  This  action,  however,  creates  a  dichotomy  between  state-mandated 
accountability  and  campus  improvement  efforts  (Aper,  Cuver,  &  Hinkle,  1990). 
Accountability-driven  assessment  responds  to  the  public’s  needs  and  concerns  and  it 
improves  communication  between  state  governments,  governing  boards,  and  institutions 
(Banta  and  Moffett,  1987).  However,  the  data  generated  fi-om  mandated  assessment 
reporting  also  can  be  beneficial  to  colleges  and  universities  for  self-review  and  self¬ 
renewal.  It  provides  opportunities  for  administrators,  faculty,  and  students  to  discuss  the 
purpose  of  their  programs  and  avenues  of  improvement  (Aper,  Cuver,  &  Hinkle,  1990). 
According  to  one  report  (El-Khavvas,  1989),  sixty-one  percent  of  institutions  surveyed 
used  assessment  results  for  program  and  curriculum  evaluation  and  changes.  El-Khawas’ 
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Study  also  revealed  that  although  assessment  activities  on  many  campuses  were  a  result  of 
external  mandates,  fifty  percent  of  those  surveyed  believed  that  assessment  will 
“  ‘significantly  improve’  ”  undergraduate  education  (p.  vii).  Institutions  that  tie  their 
assessment  results,  even  those  that  are  externally  mandated,  to  the  teaching  and  learning 
process  on  their  campuses  can  be  assured  of  both  accountability  and  improvement  (Banta, 
1993). 


Student  Outcomes  Assessment  and  College  Impact  Studies 

According  to  Alexander  Astin  (1993a)  there  are  two  traditional  approaches  to 
evaluating  the  value  of  educational  programs:  resources  and  reputational.  The  resources 
model  of  assessment  is  based  on  the  idea  that  excellence  in  education  can  be  measured  by 
how  many  resources  an  institution  possesses.  The  resources  can  be  financial  in  terms  of 
endowment,  state  fimding,  or  the  things  money  can  buy  like  libraries  or  the  physical  plant. 
Resources  can  also  include  high-quality  faculty  and  students.  The  reputational  view  of 
excellence  is  based  on  the  kind  of  reputation  an  institution  enjoys.  Astin  argues  that 
within  academic  circles,  there  is  a  myth  that  a  hierarchy  of  institutions  exists  in  which  a 
few  prestigious  universities,  such  as  Harvard,  Yale,  and  Stanford  occupy  the  top  positions. 
He  refers  to  the  reputational  view  as  a  “myth”  because  he  says  that  the  “pecking  order”  is 
not  based  on  systematic  study  or  analysis  (p.  5-6). 

Astin  has  been  critical  of  these  traditional  models  of  excellence  and  has  written  often 
about  a  third  model  he  calls  the  “talent  development”  model. 

The  fimdamental  premise  underlying  the  talent  development  concept  is  that  true 

excellence  lies  in  the  institution’s  ability  to  affect  its  students  and  faculty  favorably,  to 
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enhance  their  intellectual  and  scholarly  development,  to  make  a  positive  difference  in 
their  lives  (Astin,  1993,  p.  6-7). 

This  view,  according  to  Astin,  measures  educational  excellence  in  terms  of  “impact.”  In 
other  words,  “the  most  excellent  colleges  and  universities  are  those  that  ‘add  the  most 
value’  to  the  student’s  knowledge  and  personal  development”  (p.  7). 

The  “talent  development”  view  of  educational  excellence,  often  called  the  “value- 
added”  model,  requires  that  assessment  activities  be  based  on  outcomes  rather  than 
resources  or  reputation  (Astin,  1985;  Astin,  1993a,  Terenzini,  1989).  Student  outcomes 
assessment  is  not  a  new  concept.  Student  evaluations  have  always  been  a  part  of 
academia  since  the  first  Harvard  University  graduating  class  was  subjected  to  oral 
examinations  and  senior  declamations  (Sims,  1992,  p.  3).  However,  the  recent 
accoimtability  and  assessment  movements  have  called  for  more  comprehensive  and 
systematic  assessments  to  the  questions:  “What  do  students  get  out  of  their  educations?” 
and  “What  should  students  get  out  of  attending  college?”  By  looking  at  the  impact  of 
colleges  on  students  as  defined  by  Astin’s  talent  development  model  one  can  use  the  data 
to  establish  causal  relationships  between  the  college  environment  and  observed  student 
outcomes  or  simply  document  students’  performance  at  a  single  point  in  time  (Jacobi, 
Astin,  and  Ayala,  1987,  p.  19).  Outcomes-focused  assessment  allows  institutions  to 
compare  the  differences  between  student  talents  when  they  begin  their  educations  and  at 
the  end.  This  is  a  better  measure  of  a  university’s  effectiveness  than  traditional  measures 
that  rely  on  the  number  of  students  who  graduate  or  on  the  amount  of  money  spent 
(Halpem,  1987,  p.  6). 


30 

Student  outcomes  assessment,  as  developed  by  Astin  (1993)  and  others  (Halpem, 

1987;  Jacobi,  Astin,  and  Ayala,  1987)  describes  assessment  as  determining  the  impact  of 
college  on  students.  A  number  of  impact  studies  have  been  done  and  reviews  of  such 
studies  provided  synthesis  of  the  empirical  data.  In  1969,  Feldman  and  Newcomb, 
summarized  1,500  studies  and  reported  on  largely  affective  outcomes  such  as  “values,” 
“attitudes,”  and  “interpersonal  adjustments”  to  name  but  a  few.  Similarly,  Howard  Bowen 
(1977)  synthesized  studies  dealing  with  cognitive  development,  emotional  and  moral 
development,  as  well  as  societal  outcomes.  Pascarella  and  Terenzini  (1991)  referred  to 
the  work  of  Feldman  and  Newcomb  and  Bowen  as  “narrative,”  and  “explanatory 
literature”  (p.  9).  They  adopt  this  method  in  their  1991  book.  How  Colleges  Affect 
Students,  where  they  deal  with  cognitive  development,  personal  growth  dimensions,  and 
the  long-term  impact  of  college  on  the  quality  of  life.  These  two  authors  were  also 
concerned  about  the  implications  of  their  findings  to  state  and  federal  policy.  In  1993, 
Astin  wrote.  What  Matters  in  College?:  Four  Critical  Years  Revisited,  in  which  he 
updated  his  1 977  study  by  drawing  on  longitudinal  data  of  over  500,000  students  and 
1,300  institutions  compiled  by  the  Cooperative  Institutional  Research  Program  (CIRP). 
Astin  states  that  “[t]he  size  and  scope  of  CIRP  make  it  possible  to  employ  highly 
sophisticated  multivariate  controls  over  a  large  number  of  potentially  biasing  variables— in 
particular,  the  characteristics  of  the  entering  students  that  might  predispose  them  to  pick 
particular  types  of  colleges  or  program”  (p.  4).  Astin  seeks  to  not  only  focus  on  the 
differences  among  the  different  types  of  institutions  but  also  focuses  on  differences  in 
students’  experiences  while  they  attend  these  institutions  (p.  7). 
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These  impact  studies  reveal  that  there  has  been  significant  interest  in  how  college 
affects  students.  These  studies  deal  with  both  cognitive  and  affective  outcomes,  covering 
a  broad  range  of  student  outcomes.  Institutions  must  determine  for  themselves  what 
levels  of  outcomes  they  want  to  assess.  It  is  important  to  remember,  however,  that  no 
matter  what  institutions  assess  or  why  they  embark  on  assessment  innovations, 
institutional  improvement  is  the  underlying  reason  to  embark  on  a  campus- wide 
assessment  effort. 


Institutional  Assessment  Programs 

There  is  no  consensus  among  institutions  about  what  outcomes  to  assess  or  how  best 
to  assess  them.  Some  writers  (Ewell,  1983;  Terenzini,  1989)  state  that  institutions  must 
decide  individually  for  themselves  how  best  to  assess  student  outcomes  by  tying 
assessment  efforts  to  goals  and  objectives  of  the  institution.  Others  (Marchese,  1988) 
believe  that  the  major  question  of  assessment  should  be:  “What  do  we  do  with  the 
information?”  Regardless  of  one’s  opinions  about  the  nature  and  focus  of  assessment, 
there  are  many  examples  of  institutional  assessment  programs  in  the  literature.  While  it  is 
not  possible  to  numerate  all  of  them  here,  a  sampling  of  programs  is  helpful  so  that  the 
reader  can  see  the  complexity  and  diversity  of  student  outcomes  assessment  programs. 

Peter  Ewell  (1985)  grouped  institutional  programs  into  two  categories  that  he  calls 
“first  wave”  and  “second  wave”  programs.  The  “first  wave”  of  programs  were  early 
pioneers  in  the  assessment  movement.  These  programs  rose  gradually  in  the  1970s  and 
the  early  1980s  and  served  as  examples  to  those  that  followed.  “Second  wave” 
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institutions  are  those  that  instituted  assessment  programs  in  the  mid-1980s  as  the 
movement  gained  momentum.  Three  schools  that  fall  into  the  “first  wave”  category  and 
are  cited  often  in  the  literature  are  Alvemo  College,  the  University  of  Tennessee, 
Knoxville;  and  Northeast  Missouri  State  University. 

Alvemo  College  is  a  four-year  liberal  arts  college  for  women  in  Milwaukee,  Wisconsin 
with  a  student  population  of  about  2,500.  In  1973,  feculty  and  admimstrators  at  Alvemo 
decided  that  graduation  at  their  institution  should  be  contingent  upon  student  achievement 
in  eight  specific  areas:  communication,  analysis,  problem  solving,  valuing  in  decision 
making,  social  interaction,  global  perspectives,  effective  citizenship,  and  aesthetic 
responsiveness  (Erwin,  1991;  Loacker  and  Mentkowski,  1993;  Marchese,  1988).  At 
Alvemo  College,  students  play  a  significant  role  in  their  own  learning  and  assessment. 

This  “assessment-as-leaming”  approach  requires  that  students  collaborate  with  faculty  and 
trained  assessors  from  the  Milwaukee  business  and  professional  community  to  assess  their 
knowledge  and  abilities  as  they  progress  through  school  (Erwin,  1991).  Although 
assessment  at  Alvemo  focuses  largely  on  individual  students,  the  college  also  looks  at  the 
impact,  value,  validity,  and  effectiveness  of  Alvemo’s  programs  through  its  Office  of 
Research  and  Evaluation  (Hutchings  and  Marchese,  1 990;  Loacker  and  Mentkowski, 
1993). 

At  the  University  of  Tennessee,  Knoxville  (UTK),  legislators  tied  assessment  results  to 
funding  to  promote  institutional  improvement  and  to  respond  to  state  mandated 
requirements  for  accountability  (Santa,  1988).  In  1979,  the  Tennessee  Higher  Education 
Commission  began  an  experiment  by  instituting  a  performance  funding  feature  that 
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awarded  up  to  five  percent  of  UTK’s  annual  state  allocation  based  on  its  performance  in 
five  areas  (Banta  and  Fisher  in  Folger,  1984).  The  achievement  areas  targeted  by  UTK 
included  general  education,  the  major,  and  opinions  about  the  quality  of  academic 
programs  and  services  (Erwin,  1991;  Hutchings  and  Marchese,  1990). 

Northeast  Missouri  State  University  was  a  regional  comprehensive  college  that  became 
a  public  university.  By  1981  it  had  a  comprehensive  assessment  system  that  required 
students  to  take  a  nationally  normed  test  upon  entry  to  and  completion  of  a  general 
education  program,  a  writing  assessment,  and  a  student  survey  (Young  and  Knight  in 
Banta,  1993).  Northeast  Missouri’s  assessment  program  is  perhaps  best  known  for  its 
advocacy  of  Astin’s  “value-added”  approach  to  assessment.  Information  on 
demographics,  standardized  achievement  scores,  and  attitudinal  data  are  collected.  Data 
on  these  variables  combined  to  document  Northeast  Missouri’s  improvement  efforts  to 
satisfy  public  accountability  (Erwin,  1991). 

In  the  mid-1980s,  a  multitude  of  institutional  assessment  programs  emerged  as  calls  for 
accoimtability  increased.  The  “second  wave”  of  institutions  followed  Alvemo,  UTK,  and 
Northeast  Missouri  with  active  programs  (Ewell,  1987;  Hutchings  and  Marchese,  1990). 
These  institutions  include  Kean  College,  James  Madison  University,  King’s  College, 
Miami-Dade  Community  College,  Mt.  Hood  Community  College,  South  Dakota  State 
University,  Ohio  University,  Clayton  State  College,  and  the  State  University  of  New  York 
College  at  Plattsburgh,  to  name  but  a  few.  All  of  these  institutional  assessment  programs 
evolved  differently  but  space  does  not  permit  a  summary  of  all  of  them.  Only  the  first 
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In  1985,  Kean  College  in  New  Jersey  began  using  a  variety  of  assessment  tools  to 
promote  improvement,  as  well  as  academic  and  personal  growth  for  its  students. 
Assessment  devices  included  analyzing  student  information  data  and  test  scores, 
evaluating  portfolios  and  senior  projects,  surveys  and  exit  interviews,  focus  groups,  and 
requiring  a  variety  of  performances  specific  to  the  major  (Yoimg  and  Knight,  1993).  Kean 
College  stresses  skills,  subject  matter  knowledge,  and  human  and  professional  values  in  its 
assessment  program  (Erwin,  1991).  When  Hutchings  and  Marchese  visited  Kean  College 
in  1990  they  found  that  the  model  used  there  “minimizes  institution- wide  measures  and 
focuses  on  self-assessment  within  majors  and  programs,  some  forty  in  all,  with  each 
faculty  unit  at  liberty  to  devise  its  own  approach”  (Hutchings  and  Marchese,  1990). 

Like  Kean  College,  James  Madison  University  (JMU)  also  stresses  knowledge,  skills, 
and  human  values  in  its  assessment  program.  JMU  was  supported  by  a  $125,000  grant 
from  the  State  Coimcil  of  Higher  Education  for  Virginia,  which  resulted  in  a  collaborative 
effort  among  administrators,  faculty,  and  students,  to  develop  an  institution-wide 
assessment  program.  The  approach  used  by  JMU  was  unique  in  that  four  models  of 
assessment  were  pilot  tested  in  different  departments:  nursing  (the  Alvemo  model), 
Russian  language  department  (Northeast  Missouri  model),  economics  (the  Tennessee 
performance  model),  and  theater  (the  discrepancy  evaluation  model).  At  the  end  of  a  year, 
the  assessment  committee  proposed  seven  dimensions  of  student  assessment:  general 
education,  objectives  across  the  curriculum,  functional  skills,  affective  development, 
degree  of  student  challenge,  assessment  in  the  major,  and  alumni  follow-up  (Ewell,  1987; 
Erwin,  1991;  Hutchings  and  Marchese,  1990;  Marchese,  1988).  JMU,  like  Kean  College, 
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established  an  oflBce  of  student  assessment  and  appointed  a  full-time  director  for  its 
assessment  program  that  helped  faculty  and  administrators  design  and  analyze  assessment 
data  (Ewell,  1987;  Erwin,  1991). 

King’s  College  in  Wilkes-Barre,  Pennsylvania  is  a  private  liberal  arts  college  with  a 
full-time  student  body  of  approximately  1,750.  At  King’s,  the  assessment  program  is 
“course  embedded,”  where  assessment  is  located  within  courses,  run  by  the  faculty,  and 
focuses  on  individual  students  (Hutchings  and  Marchese,  1990).  In  1985,  King’s  College 
implemented  a  core  curriculum  that  is  required  by  everyone  and  constitutes  50%  of 
graduation  requirements.  The  core  curriculum  revolves  around  eight  “transferable”  skills 
of  critical  thinking,  creative  thinking  and  problem  solving,  writing,  oral  communication, 
quantitative  analysis,  computer  literacy,  library  and  information  technology,  and  values 
awareness  (Erwin,  1991;  Hutchings  and  Marchese,  1990).  Because  assessment  at  King’s 
is  decentralized  and  directed  by  faculty,  student  assessment  exercises  and  exams  are  part 
of  regular  course  requirements  and  are  included  in  their  final  grades.  In  this  approach, 
faculty  must  decide  what  outcomes  they  expect  and  what  criteria  they  will  use  to  measure 
them.  (Hutchings  and  Marchese,  1990).  King’s  College  also  developed  a  “rising  junior 
essay”  to  assess  writing  and  critical  thinking.  This  essay  is  graded  by  faculty  members  and 
students  are  provided  with  feedback  on  their  performance  (Ewell,  1 987). 

The  sk  institutions  highlighted  above  are  only  a  small  portion  of  examples  found  in  the 
literature.  Many  more  examples  exist  but  space  limitations  make  it  impossible  to  list  them 
all.  Perhaps  Peter  Ewell  (1985)  expressed  it  best  when  he  wrote,  “The  assessment  of 
student  outcomes  is  a  large,  complex,  and  currently  much-debated  topic-one  that  no 
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single  volume  can  expect  to  cover  adequately”  (p.  5).  However,  the  six  illustrations 
mentioned  provide  an  overview  of  the  student  outcomes  assessment  possibilities  that  exist 
and  are  colleges  that  appear  in  the  literatme  repeatedly. 

In  summary,  colleges  and  universities  have  responded  to  external  pressures  for 
accountability  in  a  variety  of  ways.  In  the  1 993  volume  of  Campus  Trends,  El-Khawas 
reported  that  three  major  types  of  improvement  efforts  were  taking  place  on  college 
campuses  throughout  the  coimtry.  While  70%  of  those  surveyed  reported  that  they 
engage  in  Total  Quality  Management  Groups  and  Procedures,  and  82%  revealed  that 
Program  Review  was  important  on  their  campuses,  fully  98%  of  respondents  claimed  that 
their  institutions  engaged  in  student  outcomes  assessment  as  an  improvement  mechanism 
(p.  16).  Clearly,  outcomes  assessment  has  become  the  principle  mechanism  by  which 
institutions  assess  their  effectiveness.  EI-Khawas  attributes  this  to  the  fact  that  accrediting 
agencies  now  expect  institutions  to  have  comprehensive  assessment  programs  tied  to 
student  outcomes  (p.  1 7).  However,  critics  of  state  mandated  assessment  claim  that 
results  are  not  being  used  for  improvement.  States  must  balance  between  being  too 
limited  or  too  inclusive  in  their  definitions  of  what  should  be  examined.  The  results  of 
assessment,  after  aU,  can  be  only  as  good  as  the  data  upon  which  they  are  based 
(McMillan,  1994). 

While  many  of  the  same  assessment  issues  exist  in  the  mid  1 990s  that  existed  in  the 
1980s,  those  involved  in  assessment  are  much  more  concerned  today  about  whether  their 
efforts  are  making  a  difference  on  campus.  Trudy  Santa’s  recent  (1993)  edited  volume 
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concludes  that  the  assessment  movement  has  instilled  new  enthusiasm  for  teaching  on 
campuses  throughout  the  country.  Not  only  have  feculty  and  staff  become  more  familiar 
with  the  literature  on  student  growth  but  they  have  begun  to  use  some  of  the 
recommendations  in  their  own  work.  According  to  Banta,  “there  are  so  many  examples  of 
assessment-  based  changes  in  the  environment  for  learning. .  .that  it  seems  quite 
appropriate  to  assume  that  student  learning  is  increasing  or  will  increase  as  the  changes 
have  time  to  achieve  their  intended  purposes”  (p.  373).  This  optimistic  viewpoint 
notwithstanding,  the  major  challenge  facing  those  involved  in  institutional  assessment 
efforts  as  the  movement  approaches  the  twenty-first  century  is  to  prove  that  assessment 
measures  are  addressing  educational  problems.  In  other  words,  not  only  wiU  educators  be 
required  to  prove  that  their  educational  programs  are  effective,  but  they  will  also  have  to 
prove  that  their  assessment  programs  work  as  well. 

Although  ACSC  is  not  a  traditional  institution  of  higher  education,  it  contains  elements 
that  make  comparisons  possible.  The  College  is  interested  in  creating  a  challenging, 
learning  environment  where  students  will  grow  intellectually.  Faculty  and  staff  at  the 
College  have  ten  months  to  do  this  as  well  as  to  assess  general  student  abilities.  Because 
of  the  short  duration  of  the  program,  overall  assessment  of  program  elements  and  how 
well  they  are  meeting  the  needs  of  their  students  becomes  central  to  improvement  efforts 
at  the  College. 
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Air  Command  and  Staff  College  and  Assessment 


The  official  mission  statement  of  ACSC  is  “to  educate  midcareer  officers  to  develop, 

advance,  and  apply  air  and  space  power  in  peace  and  war”  (Curriculum  Plan,  1995,  p.  1). 

Although  this  mission  statement  is  much  too  broad  to  express  the  totality  of  ACSC’s 

curriculum,  it  does  illuminate  the  central  focus  of  the  overall  program— the  application  of 

air  and  space  power.  The  1995  Curriculum  Plan  elaborates  on  this  mission  by  stating: 

ACSC’s  resident  curriculum  emphasizes  the  analytical  and  practical  tools 
students  need  as  fixture  military  leaders. . .  .It  remains  a  book  and 
technology-based  curriculum  exploring  the  works  of  many  great  thinkers 
and  strategists— military  and  civilian.  The  students  begin  their  studies 
addressing  the  large  conceptual  issues  of  air  campaigning  and  end  with  a 
practicum  applying  their  knowledge  of  air  and  space  power  in  a  practical 
application. 

This  challenging  educational  environment  fosters  teamwork  and  team 
building  between  faculty  and  students  and  students  themselves.  The  faculty 
helps  students  reach  higher  levels  of  creative,  analytical  thought  and  a 
deeper  mderstanding  of  the  requisites  of  command  and  the  application  of 
air  and  space  power,  (p.  1) 

To  meet  the  above  goals,  the  current  ACSC  approach  to  learning  involves  a  team 
concept.  Faculty  teams  develop  and  teach  the  curriculum,  largely  in  seminars  with 
lectures  comprising  only  1 5  percent  of  the  total  curriculum  hours.  In  addition,  through  a 
comprehensive  research  program,  student/feculty  teams  contribute  to  the  growing  body  of 
knowledge  about  aerospace  applications  and  theoretical  constructs,  while  learning  to  think 
critically  and  creatively.  According  to  the  1 995  Curriculum  Plan,  “The  college  expects 
students  and  faculty  to  chart  new  waters,  and  look  at  war  and  conflict  fi'om  the 
perspective  of  rapidly  developing  technologies,  capabilities,  and  theories”  (p.  4).  Many  of 
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these  research  projects  are  publishable  works  and  often  become  part  of  the  required 
reading  for  later  ACSC  classes.  While  research  projects  comprise  much  of  the  ACSC 
curriculum,  instructors  also  integrate  simulations,  case  studies,  and  computer  war  games 
throughout  their  courses.  A  multi-million  dollar  “space  lab,"  the  Combat  Applications 
Facility,  allows  students  and  faculty  to  become  familiar  with  space  technology  through 
hands-on  practice  and  applications.  In  addition,  every  student  receives  a  notebook 
computer  and  software  to  help  them  synthesize  information  more  efficiently.  ACSC’s 
curriculum  is  also  enhanced  through  a  distinguished  leader  guest  speakers’  program  and 
field  trips.  The  program  also  makes  use  of  a  reciprocal  exchange  program  with  the  Royal 
Air  Force  Staff  College,  the  German  Armed  Forces  Staff  College,  the  Canadian  Forces 
Command  and  Staff  College,  and  more  recently,  the  Gagarin  Military  Air  Academy  in 
Moscow,  Russia  (Curriculum  Plan,  1995). 

In  order  to  evaluate  the  courses  and  the  methods  used  at  the  College,  ACSC  evaluators 
use  a  multi-tiered  method  administered  by  the  Evaluation  Section  at  the  College.  At  the 
end  of  the  program,  students  fill  out  an  end-of-program  questionnaire.  This  questionnaire 
consists  of  seven  sections  that  include  multiple-choice,  rating  scale,  and  open-ended 
questions.  Through  a  competency  rating  scale,  students  are  asked  to  rate  themselves  on 
knowledge  and  understanding  in  each  area  of  instruction  prior  to  attending  ACSC  and 
after  completing  the  course.  In  addition,  students  rate  the  curriculum  and  presentation 
methodologies  used  at  the  College.  ACSC  evaluators  compile  and  report  the  data  from 
the  end-of-program  survey  as  statistical  descriptive  data,  consisting  largely  of  percentages. 
They  highlight  negative  trends  and  distribute  the  reports  to  the  ACSC  Commandant’s  and 
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Dean  of  Faculty’s  offices  and  to  other  interested  parties  (ACSC  Office  of  Evaluation, 
1994). 

Besides  the  end-of-program  survey,  ACSC  evaluators  also  administer  a  graduate 
survey  and  a  graduate’s  supervisor  survey.  They  mail  these  two  instruments  to  graduates 
and  their  supervisors  three  years  after  students  have  graduated  from  the  program. 
Evaluators  are  currently  working  on  a  refined  graduate  and  graduate’s  supervisor  surveys 
that  they  anticipate  administering  one  year  after  graduation  but  these  instruments  are  still 
under  revision  at  this  writing.  The  prototype  graduate  survey  that  is  being  developed  uses 
Likert-type  response  sets  to  ask  students  to  rate  ACSC’s  value  as  well  as  curriculum 
content  and  structure.  As  with  the  end-of-program  data,  graduate  survey  data  is  reported 
as  descriptive  information  and  distributed  to  key  leaders  at  the  College. 

Along  with  internal  evaluations,  external  evaluators  from  various  agencies  also  visit 
ACSC.  Two  agencies  that  annually  assess  the  programs  at  Air  University  and  ACSC  are 
the  Board  of  Visitors  (BOV)  and  the  Office  of  the  Inspector  General  (IG).  The  BOV 
convenes  at  Maxwell  Air  Force  Base  once  every  year,  usually  in  the  spring.  They  receive 
numerous  briefings,  make  observations,  and  read  detailed  reports.  Their  reports  ordinarily 
consist  of  general  comments  about  each  AU  agency  followed  by  recommendations.  The 
last  BOV  visit  was  April  10-11,  1995.  In  their  report,  the  BOV  commented  about  the 
“unique  and  exciting  military  educational  experience”  at  ACSC  and  lauded  the  College  for 
developing  “computer-based  instructional  material”  (Air  University,  1994,  p.l7).  While 
the  BOV  is  responsible  for  policy  issues  and  the  overall  value  of  PME  to  the  Air  Force, 
the  IG  inspections  focus  on  compliance  to  regulations  and  operational  readiness.  The  IG 
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inspectors,  like  the  BOV,  conduct  their  investigation  once  a  year  and  look  at  each  AU 
agency  separately.  The  IG  examiners  place  each  organization  into  one  of  five  categories: 
outstanding,  excellent,  satisfactory,  marginal,  or  unsatisfactory.  The  performance 
categories  rated  by  the  IG  inspectors  in  1995  included  supervision  and  operations, 
education  and  feculty,  students  and  support,  and  research  (Inspector  General,  1995).  The 
1995  IG  inspection  of  ACSC  rated  the  College  with  an  overall  “excellent.”  The  IG 
praised  the  Commandant  for  implementing  revolutionary  changes  in  instructional 
methodology  and  mission,  which  increased  the  “efficiency  and  quality  of  the  education, 
and  enhanced  the  school’s  reputation  as  a  world-class  practical  and  theoretical  research 
organization”  (p.  72).  While  the  IG  team  looks  at  AC  SC’s  curriculum,  they  are  mainly 
interested  in  administrative  fimctioning  and  do  not  attempt  to  assess  the  quality  of 
teaching  methods  or  pedagogy. 

In  addition  to  the  formal  methods  of  evaluation  conducted  by  ACSC  evaluators  and 
external  agencies  like  the  BOV  and  IG,  records  maintained  at  the  Air  University  library 
revealed  that  several  individuals  have  been  interested  in  the  effectiveness  of  ACSC’s 
programs  and  practices.  However,  most  of  the  studies  found  in  the  archives  were  done  as 
student  research  projects  in  partial  fulfillment  of  ACSC  requirements.  It  is  important  to 
summarize  a  few  of  these  studies  to  show  what  kind  of  evaluation  research  has  been  done 
at  ACSC  in  the  ptist. 

Several  studies  (Holmes,  1987;  Mason,  1988;  Quintanilla,  1986)  analyzed  ACSC 
graduate  surveys  in  different  years  and  reported  quantitative  data  as  percentages,  means, 
and  modes.  The  purpose  of  all  three  of  these  studies  was  to  determine  if  ACSC  was 
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accomplishing  its  mission  and  to  report  on  the  degree  of  student  satisfaction  of  ACSC’s 
program  and  curriculum.  All  three  studies  were  negative  toward  the  ACSC  research 
component  but  offered  overall  positive  comments  about  the  program.  The  descriptive 
reports,  however,  did  not  attempt  to  make  inferences  about  what  elements  of  the  program 
contributed  to  the  success  of  ACSC.  One  study  (McNally,  1979)  used  correlational 
statistics  (Kendall  Rank  Correlation  Coefficient)  to  compare  student  competencies  prior 
and  post  training  in  one  section  of  the  program,  the  Command  and  Management  Section. 
McNally  found  that  the  program  did  not  significantly  change  student  attitudes  and 
recommended  minor  changes  in  instructional  methods  at  ACSC.  One  study  (Smith,  1978) 
compared  the  instructional  methods  at  ACSC  with  those  used  in  civilian  institutions  and 
other  PME  schools.  Smith  recommended  that  the  school  retain  its  written  structure  and 
oral  critiques  but  that  it  shorten  its  evaluation  periods.  A  great  number  of  studies  (Davis, 
1991;  Griffin,  1995;  Macomber,  1966;  Orleans,  1949)  used  qualitative  and  historical 
methods  to  address  criticisms  of  ACSC  and  its  programs  and  offered  suggestions  about 
the  future  of  ACSC  and  officer  PME  in  the  Air  Force.  Davis’  historical  approach 
addressed  specific  criticisms  of  Air  University  and  its  programs,  concluding  that  they  have 
evolved  into  a  “doctrine”  of  PME.  One  of  these  studies  (Griffin,  et.al,  1995)  traced  the 
history  of  the  Air  Corps  Tactical  School  (ACTS),  the  first  Air  Force  PME  program  which 
was  phased  out  in  the  1940s.  Griffin’s  research  concluded  that  PME  should  look  at  the 
historical  precedent  set  by  ACTS  to  adequately  deal  with  the  future.  Macomber  (1966) 
and  Orleans  (1949)  used  qualitative  methods  to  evaluate  ACSC,  offering  no  statistical 
analysis.  Neither  study  found  any  serious  problems  with  teaching  methods  used  at  ACSC. 
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Macomber  suggested  changes  to  the  speaking  and  writing  programs  and  voiced  concern 
about  instructor  competency  to  teach  those  skills.  Orleans  found,  through  interviews  with 
faculty  and  students,  direct  observations,  and  examination  of  teaching  materials,  that 
although  there  was  a  positive  attitude  of  faculty  and  staff,  the  curriculum  and  texts  needed 
further  development.  A  more  quantitative  study  (Welton,  1976)  attempted  to  compare 
student  success  at  ACSC  with  other  PME  schools  and  methods.  This  study  was  interested 
in  establishing  a  relationship  between  success  at  Squadron  Officers’  School  (SOS)  and 
success  at  ACSC.  Welton  concluded  that  there  was  a  positive  relationship  between 
success  at  SOS  and  success  at  ACSC  and  that  attendance  had  a  greater  impact  on  students 
in  non  scientific  careers  than  on  those  in  scientific  careers.  Finally,  two  studies  (HammeU, 
1987;  Wiese,  1977)  were  concerned  with  the  validity  and  reliability  of  ACSC  survey 
instruments.  Both  studies  validated  the  instruments  in  use  at  ACSC.  However,  Wiese 
recommended  that  the  survey  instrument  be  revised,  while  HammeU  suggested  the 
continued  use  of  the  survey  instrument  and  methods. 

In  addition  to  the  internal  studies  summarized  above,  a  few  external  studies  were 
uncovered  as  a  result  of  this  literature  review.  Two  of  them  (Shelburne,  1953;  Dolan, 
1984)  were  dissertations  and  reflected  the  style  and  requirements  of  their  respective 
universities  and  the  time  period  in  which  they  were  writing.  Shelburne’s  dissertation 
(1953)  for  the  University  of  Chicago  was  a  qualitative  historical  report  on  the  Air  Corps 
Tactical  School.  Shelburne  concluded  that  the  relevance  of  ACTS  was  evident  in  the 
World  War  II  experience  and  that  advanced  training  was  justified  to  sustain  the  Air 
Force’s  capability.  Dolan’s  1984  dissertation  for  the  University  of  Southern  California 
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was  an  analysis  of  ACSC’s  nonresident  correspondence  program  (now  the  distance 
learning  program).  This  quantitative  study  revealed  that  correspondent  graduates  felt  that 
ACSC’s  utility  was  limited  as  compared  to  resident  graduates.  Dolan  recommended  that 
more  emphasis  be  placed  on  the  professional  development  aspect  of  the  correspondence 
program  to  compensate  for  the  differences  in  utility  scores.  Two  additional  external 
studies  (Getzels  and  Cuba,  1956;  Davis  and  Donnini,  1991)  stand  out  in  the  literature. 
Egon  Cuba,  who  is  perhaps  most  well  known  for  his  advocacy  of  qualitative  methodology 
and  naturalistic  inquiry,  collaborated  with  Jacob  Getzels  in  1956  to  conduct  a  study  titled 
“Role  Conflict  and  Instructor  Effectiveness  at  the  Air  Command  and  Staff  School.”  This 
study  was  sponsored  by  the  Air  Force  Personnel  and  Training  Research  Center  and  was 
the  first  report  in  a  broader  research  project.  Guba  and  Getzels  studied  the  effect  of  role 
conflict  on  teaching  effectiveness  at  ACSC,  postulating  the  theory  that  the  possible 
inconsistency  among  the  various  roles  that  an  officer  must  play  while  on  faculty  would 
affect  job  performance.  The  study  found  that  there  is  a  relationship  between  role  conflict 
and  teaching  effectiveness,  with  high  conflict  scores  correlating  significantly  with  teaching 
ineffectiveness.  Although  this  study  was  done  almost  forty  years  ago,  it  is  the  only 
examination  of  ACSC  conducted  by  a  well-known  researcher  and  educator  and  the  only 
investigation  that  specifically  focuses  on  faculty  effectiveness.  Finally,  research  by  Davis 
and  Donnini  resulted  in  a  1991  book  titled,  Professional  Military  Education  for  Air  Force 
Officers:  Comments  and  Criticisms.  Both  authors  were  active-duty  Air  Force  officers 
assigned  to  the  Air  University  Center  for  Aerospace  Doctrine,  Research,  and  Education. 
Although  this  book  did  not  focus  solely  on  ACSC,  it  is  an  important  monograph  because  it 
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traces  the  history  of  Air  Force  PME  between  1946  and  1987.  Davis  and  Donnini 
examined  over  345  documents-letters,  regulations,  manuals,  studies,  reports,  catalogs, 
histories— and  summarized  the  history  of  Air  Force  PME  as  seen  through  the  eyes  of  its 
critics. 

The  above  studies  distilled  from  the  literature  add  to  data  available  about  the 
effectiveness  of  ACSC  and  its  programs.  However,  it  is  impossible  to  know  the  extent  to 
which  these  studies  have  contributed  to  changes  in  ACSC’s  curriculum  without  ftirther 
investigation.  Utilization  of  evaluation  results  is  a  continuing  issue.  Although  ACSC 
evaluators  compile  and  disseminate  information  to  key  leaders  at  the  College,  there  are  no 
measures  in  place  to  link  improvement  efforts  to  evaluation  results.  It  is  even  less 
doubtfiil  that  research  projects  such  as  those  summarized  above  are  used  to  any  extent  as 
they  are  filed  away  in  the  archives  after  completion. 

In  summary,  assessment  methods  currently  used  at  ACSC  only  provide  stake  holders 
with  descriptive,  anecdotal,  and  largely  subjective  information.  Research  does  not  go  far 
enough  to  make  predictions  about  what  areas  of  the  program  affected  student  competence 
on  outcome  variables.  While  assessment  efforts  at  the  College  look  at  student  outcomes 
to  some  extent,  they  do  not  offer  targeted  evidence  about  why  certain  methods  work  while 
others  do  not.  This  dissertation  went  beyond  descriptive  information  and  filled  a  gap  in 
the  literature  by  indicating  those  connections.  This  will  allow  stake  holders  at  ACSC  to 
pinpoint  the  strength  and  weaknesses  of  their  program,  moving  their  assessment  efforts 
beyond  accountability  and  into  program  improvement. 
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CHAPTER  THREE 

METHODOLOGY 

Overall  Research  Design 

This  study  adapted  the  input-environment-output  (I-E-0)  model  developed  by 
Alexander  Astin  and  detailed  in  his  1993  book,  Assessment  for  Excellence  (first  published 
in  1991).  Astin  states  that  “the  basic  purpose  of  assessing  students  is  to  enhance  their 
educational  development”  (p.  4).  The  I-E-0  model  provides  information  about  how 
different  educational  activities  and  practices  affect  outcomes,  leading  to  program  and 
faculty  improvement  (Astin,  p.  37).  The  researcher  can  examine  each  component  of  the 
model  separately  --inputs,  environment,  and  outcomes-  but  according  to  Astin,  it  is 
important  to  always  evaluate  outputs  in  terms  of  inputs  to  get  an  accurate  picture  of  how 
effective  the  educational  program  was  in  developing  student  talents.  Astin  defines  inputs 
as  the  personal  qualities  that  students  bring  with  them  to  the  educational  program; 
environment  consists  of  the  students’  experiences  during  the  program;  and  the  outcomes 
refer  to  the  talents  that  the  program  intends  to  develop  in  its  students  (p.  1 8).  In  research 
design  terms,  input  and  environmental  variables  are  the  independent  variables,  and  the 
outcome  variables  are  the  dependent  variables.  The  I-E-0  design  allows  the  researcher  to 
control  for  potentially  biasing  input  variables,  therefore,  getting  a  more  accurate  estimate 
of  the  effects  of  different  environments  on  outcomes  (p.  19). 
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The  major  research  question  of  this  study  was:  “How  effective  was  Air  Command  and 
Staff  College  in  meeting  its  goals  and  what  contributed  to  this  effectiveness  as  perceived 
by  the  students  who  graduated  from  the  program?”  In  this  study,  effectiveness  was  the 
extent  to  which  students  perceived  their  competencies  in  student  outcome  goals.  ACSC’s 
outcome  goals  were  defined  as  those  things  that  the  program  purports  to  instill  in  its 
students.  The  researcher  used  a  90-item  questionnaire  called  “Student  Perceptions  of 
Program  Effectiveness  Questionnaire,”  to  collect  data  for  the  study.  The  survey  asked 
students  to  rate  the  quality  of  the  teaching  methods  and  program  activities  at  ACSC  and  to 
assess  their  competency  on  selected  student  outcome  measures  (see  Appendix  A  for  the 
questionnaire).  The  researcher  then  analyzed  the  relationships  between  the  inputs, 
environment,  and  outcomes  at  Air  Command  and  Staff  College  and  identified  which 
environmental  variables  were  most  predictive  of  students’  perceived  competence  on 
outcome  variables.  Descriptive  data  were  analyzed  and  reported  using  means,  standard 
deviations,  and  percentages.  Relational  data  were  reported  using  cross  tabulations  and 
Pearson  product-moment  correlation  coefficients  (Pearson  “r”s).  Finally,  multiple 
regression  analysis  was  used  to  control  for  potentially  biasing  input  data  so  that  the 
researcher  could  make  predictions  about  which  environmental  variables  most  affected 
student  outcomes. 


Study  Institution 

The  institution  chosen  for  this  study  was  Air  Command  and  Staff  College,  a  major  Air 


Force  professional  military  education  program  located  at  Maxwell  Air  Force  Base  in 
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Montgomery,  Alabama.  Students  are  chosen  to  attend  ACSC  in  conjunction  with  their 
selection  to  the  rank  of  Major  at  about  the  ten-year  point  in  their  careers.  Although 
attendance  at  the  College  is  not  mandatory,  those  who  are  chosen  and  decline  to  attend 
may  jeopardize  their  opportunity  for  future  promotions.  The  researcher  chose  Air 
Command  and  Staff  College  to  evaluate  because  of  its  importance  in  developing  Air  Force 
leaders  who  will  be  affecting  Air  Force  policy  for  the  next  ten  to  twenty  years.  For  the 
Academic  Year  (AY)  1994-1995  class,  ACSC’s  curriculum  consisted  of  nine  academic 
courses  (for  a  complete  description  of  these  courses,  please  see  Appendix  B).  These 
courses  were  grouped  into  three  different  departments  within  the  ACSC  structure: 
Command/Strategic  Studies  Department,  War/Theater  Level  Studies  Department,  and 
War  Theory/Campaign  Studies  Department.  Besides  these  major  academic  components. 
For  AY  1994-1995,  the  ACSC  structure  also  supported  a  Distance  Learning  Department, 
a  Joint  Warfare  Studies  Department,  an  Academic  Support  Department,  a  Wargaming 
Department  and  an  Evaluation  Component  (Curriculum  Plan,  1995). 

Population  and  Study  Group 

In  addition  to  active  duty  Air  Force  officers,  each  ACSC  class,  consisting  of  close  to 
600  students,  also  hosts  selected  US  civilians  (approximately  100),  international  officers 
(approximately  80)  from  several  different  countries,  and  exchange  officers  from  the  other 
services  (ACSC  Curriculum  Plan,  1995;  Air  University  Catalog,  1995).  Each  class  begins 
in  August  and  graduates  the  following  June,  a  total  of  ten  months.  For  AY  1994-1995, 
the  ACSC  graduating  class  began  with  584  members  including  civilians,  international 
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officers,  and  exchange  officers.  However,  the  researcher  was  only  interested  in  measuring 
the  perceptions  of  active  duty  Air  Force  officers.  This  subpopulation  consisted  of  395 
students.  These  officers  graduated  in  June,  1995  and  returned  to  their  Air  Force  jobs  at 
bases  located  in  the  United  States,  Europe,  Panama,  and  the  Far  East.  Mailing  labels  were 
obtained  from  the  Air  Force  Personnel  Center  at  the  end  of  September  and  questionnaires 
were  mailed  to  the  entire  subpopulation  on  November  3, 1995.  The  five  months  between 
graduation  and  the  first  mailing  provided  ample  time  for  participants  to  move  to  new 
locations  and  settle  into  their  new  jobs. 

Out  of  the  original  395  questionnaires  sent,  228  usable  questionnaires  were  returned. 
These  figures  represent  a  return  rate  of  58  percent.  However,  after  accoimting  for  the 
first  and  second  mailing,  107  questionnaire  packets  were  returned  as  “undeliverable  as 
addressed”  mail.  Of  those  107  packets,  75  (70  percent)  were  returned  from  Air 
Command  and  Staff  College.  Although  some  graduates  remained  at  ACSC  as  feculty  and 
staff  members,  this  large  number  of  returned  ACSC  mail  indicates  bad  addresses  rather 
than  a  postal  delivery  problem.  One  possible  reason  for  the  number  of  bad  addresses  at 
ACSC  is  the  number  of  graduates  who  received  their  assignments  late.  Another  reason 
may  be  that  some  graduates  went  to  follow-on  training  or  schools  which  are  considered 
temporary  duty.  In  both  situations,  graduates’  last  known  assignment  and  work  address 
would  be  carried  in  personnel  records  until  they  arrived  at  their  permanent  location.  While 
it  is  possible  to  trace  individuals  within  the  personnel  system,  the  researcher  did  not  feel 
that  this  would  be  necessary.  There  is  no  reason  to  believe  that  the  107  individuals  who 


could  not  be  reached  because  of  bad  addresses  were  different  from  those  who  returned 
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questionnaires.  In  other  words,  the  fact  that  these  individuals  could  not  be  contacted  did 
not  have  a  significant  impact  on  survey  responses  and  did  not  skew  the  results  of  this 
study.  However,  if  one  deducts  the  107  bad  addresses  from  the  total  number  of 
questionnaires  sent  (395),  the  response  rate  increases  to  78  percent  (78  %). 

Research  Questions 

The  major  question  this  study  addressed  was  an  evaluation  question. 

How  effective  was  ACSC  in  meeting  its  goals  and  what  contributed  to  this 
effectiveness  as  perceived  by  the  students  who  graduated  from  the  program? 

This  question  revolved  around  the  overall  effectiveness  of  ACSC.  Effectiveness  was  the 

extent  to  which  students  rated  their  competencies  on  student  outcome  variables.  The 

question  also  involved  determining  if  students’  perception  of  the  quality  of  environmental 

variables  affected  their  perceived  competencies  on  outcome  goals. 

To  answer  the  above  question,  the  researcher  divided  the  subsidiary  questions  into 

four  categories:  descriptive,  relational,  predictive,  and  implications.  First,  specific 

descriptive  questions  were: 

1.  What  were  the  demographic  factors  and  student  expectations  (inputs)  of  ACSC 
graduates  in  academic  year  1995? 

2.  How  did  students  rate  the  quality  of  the  teaching  methods  and  program  activities 
(environmental  factors)  at  Air  Command  and  Staff  College? 

3.  How  did  students  rate  their  competency  on  outcome  measures  at  the  time  they 
completed  Air  Command  and  Staff  College? 

Second,  the  researcher  wanted  to  know  the  relationships  between  the  independent  and 
dependent  variables.  Specific  correlational  questions  were: 
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4.  Was  there  a  relationship  between  input  variables  (demographics  and  student 
expectations)  and  student  outcomes? 

5.  Was  there  a  relationship  between  environmental  variables  (teaching  methods  and 
program  activities)  and  student  outcomes? 

Third,  the  researcher  wanted  to  make  predictive/causal  claims  about  the  relationship 
between  environmental  factors  and  outcomes.  To  ensure  that  claims  made  about  the 
effect  of  certain  environmental  factors  on  outcomes  were  accurate,  the  researcher 
“controlled”  for  other  variables  that  might  affect  outcomes.  Since  input  variables  may 
affect  the  relationship  between  environmental  variables  and  outcomes,  the  next  question 
was: 

6.  Holding  the  input  variables  (demographics  and  student  expectations)  constant, 
was  there  a  relationship  between  environmental  variables  (teaching  methods  and 
program  activities)  and  student  outcomes? 

Finally,  based  on  the  quantitative  data  compiled  from  the  questions  above  and  from  open- 
ended  questions  posed  to  academic  year  1 995  graduates,  the  researcher  made 
recommendations  about  program  changes  and/or  improvements  to  senior  ACSC  leaders 
and  faculty  members.  The  final  question  addressed  this  issue: 

7.  What  were  the  implications  of  the  results  of  this  study  for  Air  Command  and 
Staff  College? 


Rationale  for  Chosen  Variables 

To  understand  fully  the  rationale  for  the  variables  chosen  for  this  study,  it  is  necessary 
to  describe  the  student  selection  process  at  ACSC.  ACSC  students  meet  a  special 
selection  board  in  conjunction  with  their  Majors’  promotion  board  that  centrally  convenes 
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at  the  Air  Force  Personnel  Center  at  Randolph  Air  Force  Base,  Texas  each  year.  An 
officer’s  demonstrated  potential  for  key  field  grade  (Majors  and  above)  command  and  staff 
assignments  is  the  criterion  used  for  selection  to  ACSC.  No  one  at  the  College—  faculty, 
staff,  or  senior  leaders,  is  responsible  for  selecting  students  (Air  University  Catalog, 

1994).  Very  few  officers  turn  down  the  opportunity  to  attend  ACSC  in  residence.  Those 
who  decline  to  attend  ACSC  in  residence  do  so  “with  prejudice,”  which  gravely 
jeopardizes  their  future  promotion  potential. 

This  discussion  about  the  selection  process  was  presented  to  illustrate  the  difference 
between  incoming  students  to  ACSC  and  students  who  attend  traditional  institutions  of 
higher  education.  This  presents  a  unique  methodological  problem  for  the  researcher  when 
it  comes  to  determining  which  input  and  environmental  variables  to  choose  for  the  study. 
Since  there  are  no  pretest  measures  or  incoming  grade  point  averages  or  GRE  scores  to 
consider,  the  researcher  must  determine  which  input  variables  should  be  used.  Similarly, 
choosing  environmental  variables  at  ACSC  also  presents  a  methodological  problem  for  the 
researcher.  In  traditional  educational  institutions,  students  are  fi-ee  to  choose  their  courses 
of  study  or  to  determine  how  long  or  to  what  extent  they  will  be  involved  in  pursuing  their 
education.  The  ACSC  student  does  not  have  such  choices.  All  ACSC  students  attend  the 
same  courses  in  the  same  sequence.  For  the  most  part,  they  are  exposed  to  the  same 
pedagogical  techniques  presented  by  the  same  faculty  members.  On  the  surface,  it  appears 
as  if  aU  ACSC  students  have  the  same  experiences  while  participating  in  the  program. 
However,  it  is  unlikely  that  students  perceived  the  program  activities  in  exactly  the  same 
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way  nor  were  their  attitudes  toward  the  learning  environment  identical.  For  this  reason, 
the  researcher  used  student  perceptions  as  a  means  of  data  collection. 

The  following  discussion  about  the  variables  used  to  evaluate  ACSC  provides  a 
rationale  and  framework  for  the  overall  study  and  places  them  in  context  with  the  I-E-0 
model. 

Input  Variables.  Input  variables  were  those  characteristics  that  students  brought  with 
them  to  the  program.  This  study  focused  on  two  types  of  input  characteristics, 
demographic  data  and  student  expectations.  Although  the  College  currently  collects  and 
reports  demographic  data  on  each  incoming  class,  they  do  not  look  at  the  relationship 
between  demographic  data  and  student  outcomes.  The  researcher  believed  that  it  was 
important  to  know  if  ACSC  was  meeting  the  needs  of  all  of  its  students.  For  example,  it 
was  not  enough  to  know  that  seventeen  percent  of  the  students  were  female,  but  a  more 
pertinent  question  was  “What  was  the  relationship  between  gender  and  student 
outcomes?”  In  addition,  by  controlling  for  the  input  bias  of  demographic  variables 
through  multiple  regression,  the  researcher  was  able  to  predict  with  greater  certainty  the 
environmental  variables  that  most  affected  students’  perceptions  of  competence  on 
outcomes.  Operational  definitions  and  categories  of  input  variables  are  as  follows: 

Demographic  data.  It  was  important  to  know  if  ACSC  met  the  needs  of  all  of  its 
students.  Demographic  variables  were  important  determiners  of  the  College’s  ability  to 
reach  all  members  of  their  student  body.  The  following  variables  made  up  the 
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demographic  data  (please  refer  to  questions  82-87,  Section  “D”  of  the  “Student 
Perceptions  of  Program  Effectiveness”  Questionnaire  at  Appendix  A): 

a.  Age:  Although  ACSC  students  were  close  in  age,  the  researcher  was  interested  in 
how  age  affected  their  perceived  competence  on  outcome  variables.  The  age  variable  was 
divided  into  four  categories  and  perceived  student  perceptions  of  competence  on 
outcomes  were  compared  across  age  categories. 

b.  Gender:  Given  the  predominance  of  male  officers  over  female  officers  in  the  Air 
Force  as  a  whole,  the  researcher  was  certain  that  ACSC  students  were  predominantly 
male.  However,  by  looking  at  differences  in  perceived  competence  across  gender  lines, 
the  researcher  provided  senior  ACSC  leaders  with  valuable  information  about  how  well 
their  program  met  the  needs  of  its  female  officers  as  well  as  identified  program  activities 
that  contributed  most  to  their  perceived  competence  on  outcomes. 

c.  Race:  Because  of  Air  Force  demographics,  the  researcher  was  certain  that  the 
racial  makeup  of  ACSC  students  was  primarily  Caucasian.  What  was  unknown  was  how 
well  the  methods  and  processes  used  at  ACSC  affected  the  perceived  competence  of 
minority  students.  By  including  race  as  a  demographic  variable,  this  issue  was  addressed. 

d.  Career  Field:  ACSC  students  came  fi’om  a  variety  of  career  fields  with  different 
backgrounds  and  experiences.  The  researcher  was  interested  in  how  students’  career 
fields  affected  their  perceived  competence  and  what  elements  of  the  program  best  met  the 
needs  of  the  majority  of  Air  Force  specialties.  This  is  important  because  ACSC  wants  to 
provide  an  educational  environment  that  not  only  addresses  the  technical  aspects  of 
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military  service.  The  professional  military  education  of  mission  support,  logistics,  and 
medical/legal  ofiBcers  was  just  as  important  as  that  of  pilots  and  navigators. 

e.  Master  Degree  Concentration:  By  the  time  officers  reach  the  rank  of  Major,  the 
Air  Force  expects  that  they  will  have  earned  Master  Degrees.  Obviously,  different 
educational  backgrounds  provide  students  with  different  mindsets  and  opinions.  For 
example,  engineers  do  not  see  the  world  in  the  same  way  as  political  scientists.  The 
researcher  wanted  to  know  how  Master  Degree  concentration  and  Education  Level 
affected  students’  perceived  competence  and  which  elements  of  ACSC’s  program  best  met 
the  needs  of  students  from  different  educational  backgrounds. 

f  Commission  Source:  There  are  three  major  ways  in  which  Air  Force  officers  can 
receive  commissions:  Reserve  Officer  Training  Corps  (ROTC),  United  States  Air  Force 
Academy  (USAFA),  and  Officer  Training  School  (OTS).  Each  avenue  provides  different 
educational  backgroimds  and  pedagogical  experiences.  Students’  perceptions  will 
undoubtedly  vary  depending  on  their  commission  source.  By  looking  at  those  differences, 
the  researcher  was  able  to  pinpoint  program  elements  that  provided  the  best  learning 
environment  and  pedagogy  for  the  majority  of  students. 

Student  expectations  were  measured  by  presenting  students  with  a  list  of  choices  from 
which  they  were  to  choose  aU  of  the  expectations  that  applied  to  them.  The  following  is  a 
list  of  student  expectation  choices  (question  88,  Section  “E”  of  the  questionnaire  at 
Appendk  A): 

1 .  ACSC  would  improve  my  chances  for  future  promotions. 

2.  ACSC’s  curriculum  would  make  me  a  better  Air  Force  officer. 

3.  I  would  meet  other  people  and  learn  about  other  career  fields. 

4.  I  would  improve  my  golf  game. 
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5.  I  would  get  to  spend  more  time  with  my  family. 

6.  I  would  spend  a  great  deal  of  time  socializing. 

7.  I  would  learn  very  little  while  a  student  at  ACSC. 

8.  I  would  be  academically  challenged  while  a  student  at  ACSC. 

9.  Other  (please  specify) 

The  list  of  expectations  was  compiled  as  a  result  of  anecdotal  data  from  students  and 
expressed  stake  holder  concerns.  ACSC  students  have  been  in  the  Air  Force  for  several 
years  and  have  been  exposed  to  other  professional  military  education  experiences.  Over 
the  years,  they  have  acquired  a  set  of  assumptions  about  what  the  ACSC  experience 
would  do  for  them.  For  example,  anecdotal  information  revealed  that  many  oflScers  view 
ACSC  as  a  time  to  relax  away  from  their  jobs  and  spend  time  with  their  families  or  play 
golf.  Attitudes  such  as  these  may  affect  students’  perceptions  of  their  competence  on 
outcomes.  Senior  leaders  at  the  College  wanted  to  understand  what  expectations  students 
possessed  when  they  entered  the  program  and  how  these  expectations  related  to  student 
competence  on  outcomes. 

Environmental  Variables.  Environmental  variables  were  those  things  in  the  educational 
environment  that  students  experienced  while  at  ACSC.  As  already  stated,  environmental 
variables  at  ACSC  did  not  vary  considerably.  Students  may  have  experienced  different 
things  as  a  result  of  being  assigned  a  morning  versus  an  afternoon  seminar  or  they  might 
have  been  exposed  to  different  material  because  of  their  faculty  instructor’s  work 
experiences.  For  all  intents  and  purposes,  however,  ACSC  students  were  exposed  to  very 
similar  learning  environments.  What  did  vary  in  the  environment  were  student 
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perceptions.  The  thirty-two  items  that  made  up  Section  “C”  on  the  questionnaire  (items 
50-81)  were  grouped  into  the  following  six  categories  of  environmental  variables. 

Teaching  methods.  ACSC  uses  primarily  two  types  of  teaching  method;  lecture  (in- 
house  and  outside  lecturers),  and  seminars  (informal  lectures  and  colloquia-style  forums). 
Items  50-53,  Section  “C”  of  the  questionnaire  refer  to  teaching  methods  (Appendix  A). 

Technology.  The  use  of  computers  has  become  central  to  ACSC’s  overall  teaching 
methodology.  Students  use  their  laptop  computers  to  access  their  daily  schedules, 
communicate  through  electronic  mail,  and  do  much  of  their  reading  from  on-line  articles. 
In  addition,  ACSC  extensively  uses  an  interactive  computer  software  called  Toolbook©  to 
supplement  readings  and  to  aid  students  in  their  research  efforts.  Items  54-58,  74,  and  81, 
Section  “C”  of  the  questionnaire  refer  to  Technology  (see  Appendix  A). 

Curricultun.  ACSC’s  current  curriculum  is  book-based,  consisting  of  over  100  books 
and  supplemented  by  current  articles.  It  was  important  to  know  how  effective  this 
approach  was  in  fecilitating  student  learning.  Besides  curricular  content,  of  equal  value 
was  the  structure  (curricular  flow,  course  lengths,  etc.).  Items  59-62,  67,  75,  and  76, 
Section  “C”  of  the  questionnaire  refer  to  Curriculum  (see  Appendix  A). 

Research.  The  Research  component  at  ACSC  is  an  important  element  of  the  College’s 
curriculum.  ACSC  leaders  wanted  to  know  to  what  extent  this  approach  was  effective  for 
their  students.  Items  63,  68,  69,  and  77,  Section  “C”  of  the  questionnaire  refer  to 
Research  (see  Appendbc  A). 

Faculty.  Because  faculty  members  receive  evaluations  from  their  supervisors  and  from 
student  critiques,  this  study  did  not  attempt  to  evaluate  ACSC  feculty  members  to  any 
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length.  However,  the  researcher  wanted  to  know  how  students’  perceptions  of  the  quality 
of  ACSC  faculty  related  to  their  perceptions  about  their  competence  on  outcomes.  Items 
64,  65,  70,  78,  and  79,  Section  “C”  of  the  questionnaire  refer  to  Faculty  (see  Appendix 
A). 

Grading.  Before  1994,  the  grading  system  at  ACSC  was  pass/fail.  Currently,  students 
receive  letter  grades  for  their  work.  Since  anecdotal  evidence  revealed  that  students  were 
unhappy  with  the  new  grading  system,  the  researcher  added  it  as  an  environmental 
variable.  Items  66,  71,  72,  73,  and  80,  Section  “C”  of  the  questionnaire  refer  to  Grading 
(see  Appendix  A). 

The  above  sk  categories  of  environmental  variables  used  in  this  study  were  chosen  as  a 
result  of  formal  and  informal  discussions  and  interviews  with  members  of  the  faculty  and 
staff  at  ACSC.  Stake  holders  at  the  College  determined  that  these  items  were  central  to 
their  program  and  wanted  to  know  how  these  items  affected  student  competence  on 
outcomes.  Students  were  asked  to  rate  the  quality  of  program  elements  at  ACSC  using  a 
five-point  rating  scale.  The  items  contained  in  Section  “C”  corresponded  to  the 
environmental  variables  listed  above  in  the  following  manner: 

Variable  Questions 

Teaching  Methods  50,  51,  52,  53 

Technology  54,  55,  56,  57,  58,  74,  81 

Curriculum  59,  60,  61,  62,  67,  75,  76 

Research  63,  68,  69,  77 


Faculty 


64,  65,  70,  78,  79 
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Grading  66,  71,  72,  73,  80 

Students  rated  each  question  from  one  to  five  (1-5)  and  a  score  for  each  variable  v^as 
compiled.  Note  that  items  75-81  are  not  on  the  same  five-point  scale  as  the  other  items. 
Rather,  they  were  grouped  into  discrete  categories  and  were  intended  to  be  reported  as 
descriptive  information  only.  They  were  not  used  in  the  correlational  analyses  or  the 
regression  equations.  For  the  relational  and  predictive  analyses,  the  individual  items  that 
made  up  each  environmental  variable  were  added  together  to  create  “collapsed”  values. 
For  example,  the  Teaching  Methods  variable  was  created  by  adding  together  items  50-53. 
Correlations  between  environmental  and  outcome  variables  were  reported  using  Pearson 
“r”s  based  on  collapsed  values.  However,  for  the  convenience  of  the  reader,  correlations 
for  each  item  that  made  up  the  collapsed  values,  were  reported  in  table  format  and  are 
found  in  Appendix  D.  Collapsed  values  for  variables  were  also  used  in  the  regression 
analysis.  Descriptive  data  were  reported  on  each  item  individually  rather  than  on 
collapsed  values. 

Outcome  Variables.  Outcome  variables  were  the  talents  that  the  program  claimed  to 
instill  in  its  students.  The  ofScial  mission  of  ACSC  is  “to  educate  midcareer  officers  to 
develop,  advance,  and  apply  air  and  space  power  in  peace  and  war”  (Curriculum  Plan, 
1995,  p.  1).  This  mission  statement  was  too  broad  to  be  of  any  use  in  determining 
measurable  student  outcomes.  The  researcher,  faced  with  the  problem  of  what  outcomes 
to  measure,  distilled  information  from  the  nine  course  syllabi  through  a  content  analysis. 
The  content  analysis  involved  collecting  syllabi,  extracting  stated  goals  from  them,  and 
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putting  them  in  “like”  categories.  This  method  yielded  forty-four  items  (questions  5-49  on 
the  questionnaire)  that  the  College  hoped  to  instill  in  its  students.  These  items  fell  into 
one  of  four  categories  of  outcome  variables  which  are  defined  as  follows: 

Joint  Campaign  Planning  Process.  Joint  campaigns  are  military  operations  that  involve 
more  than  one  branch  of  the  service.  Because  many  military  theorists  and  strategists 
believe  that  modem  warfere  involves  cooperation  and  coordination  among  all  the  services, 
a  major  objective  at  ACSC  is  to  teach  students  about  joint  doctrine  and  war  applications. 
Items  35-45,  Section  “B”  on  the  questionnaire  were  designed  to  gather  student 
perceptions  on  how  competent  they  were  in  this  area  (see  Appendbc  A). 

Air/Space  Power  Applications.  Although  joint  campaigns  will,  more  than  likely,  be 
commonplace  in  modem  warfare.  Air  Force  officers  must  learn  the  theories,  uses,  and 
limitations  of  aerospace  power.  Future  air  campaign  planners  must  understand  how  to 
employ  airpower  as  part  of  the  overall  operation  while  working  in  concert  with  the  other 
services.  Items  46-49,  Section  “B”  on  the  questionnaire  were  designed  to  gather  data  on 
student  competence  on  this  variable  (see  Appendbc  A). 

Command  and  Leadership.  One  goal  of  ACSC  is  to  mold  mid-level  officers  into 
commanders.  The  Command  and  Leadership  variable  deals  with  commanders’  roles  and 
responsibilities  (i.e.,  people  skills,  administrative  duties,  etc.)  and  teaches  students  about 
the  unique  leadership  role  of  a  military  professional.  The  Command  and  Leadership  items 
of  the  questionnaire  are  5-16,  Section  “B”  (see  Appendbc  A). 

Critical  Thinking.  One  of  the  major  threads  running  through  all  the  courses  at  ACSC 
was  critical  thinking.  At  ACSC,  a  person  who  thinks  critically  can  use  concepts,  theories. 
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and  principles  of  war  to  solve  problems  and  make  decisions.  In  other  words,  critical 
thinking  involves  using  problem-solving  skills  to  synthesize  concepts,  applying  them  in 
new  and  different  situations.  Items  1 7-34,  Section  “B”  of  the  questionnaire  cover  the 
Critical  Thinking  variable  (see  Appendbc  A), 

As  was  the  case  with  environmental  variables,  outcome  variables  were  also  based  on 
“collapsed”  values.  For  example,  items  35-45  comprise  the  Joint  Campaign  outcome 
variable  and  are  added  together  to  create  that  variable.  Relational  and  predictive  analyses 
were  based  on  collapsed  values  for  the  most  part.  Descriptive  information,  however,  was 
reported  on  each  item  individually  to  provide  more  detail.  Questions  that  comprise 
Section  “B”  on  the  questionnaire  corresponded  to  the  above  variables  in  the  following 
manner: 


Variable 

Questions 

Joint  Campaign  Planning  Process 

35-45 

Air/Space  Power  Applications 

46-49 

Command  and  Leadership 

5-16 

Critical  Thinking 

17-34 

Students  were  asked  to  respond  to  the  question:  “How  would  you  rate  your  competency 
on  the  following  items  at  the  point  in  time  when  you  completed  Air  Command  and  Staff 
College  (see  Appendbc  A,  Section  B,  Questions  5-49)?”  Using  a  five-point  rating  scale, 
students  rated  their  competency  on  outcomes.  Notice  that  students  were  not  asked  to 
make  connections  between  their  competency  and  ACSC,  Causal  connections  between 
program  elements  and  student  perceived  competency  on  outcomes  were  made  through 
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multiple  regression  analysis  based  on  data  collected  from  the  questionnaire.  Because  of 
the  nature  of  the  design  of  this  study  and  the  researcher’s  time  frame  for  completion,  it 
was  not  possible  to  ascertain  student  competency  on  outcome  measures  prior  to  their 
attendance  at  ACSC.  Therefore,  this  study  did  not  include  any  pretest  measures  of 
student  competency  as  input  data.  Pretest  measures  would  have  provided  data  that  could 
be  compared  to  posttest  data  and  then  used  to  make  inferences  about  the  overall 
effectiveness  of  ACSC  based  on  a  pretest/posttest  design.  This  study,  however,  was  a 
cross  sectional  study  that  was  concerned  with  choosing  variables  that  would  optimize 
prediction.  In  other  words,  the  researcher  chose  the  input  variables  (demographics  and 
student  expectations)  that  were  expected  to  have  the  most  effect  on  students’  perceptions 
of  their  competence  on  outcomes.  Given  the  design  of  this  study  and  the  constraints 
placed  on  the  data  by  the  researcher,  the  data  showed  which  combination  of  variables 
(input  and  environmental)  best  predicted  students’  perceptions  of  their  competence  on  the 
aforementioned  outcome  variables. 

Instrumentation  and  Measurement 

The  researcher  developed  a  “Student  Perceptions  of  Program  Effectiveness 
Questionnaire”  which  was  mailed  to  all  395  Air  Force  officer  students  from  the  AY  1 994- 
1995  graduating  class  on  November  3,  1995  (please  refer  to  Appendix  A).  The 
questionnaire  contained  six  sections.  Section  “A”,  “Overall  Effectiveness,”  consisted  of 
four  questions  required  by  Air  University,  AC  SC’s  parent  organization.  These  four 
questions  were  “quality”  questions  referring  to  Total  Quality  Management  principles 
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adopted  by  the  Air  Force.  The  questions  in  Section  “A”  did  not  directly  correspond  to 
any  of  this  study’s  variables  and  data  from  this  section  were  not  reported.  This 
information,  however,  could  be  used  by  ACSC  as  descriptive  data  in  compliance  with  Air 
University’s  guidelines.  Section  “B”,  “Course  Objectives,”  was  designed  to  gather  data 
on  students’  perceptions  about  their  competency  on  outcome  measures  (outcome 
variables).  Section  “C”,  “Program  Activities  and  Teaching  Methods,”  asked  students  to 
rate  the  quality  of  program  activities  and  teaching  methods  (environmental  variables). 
Section  “D”,  “Demographic  Data,”  asked  students  to  report  their  individual  backgrounds 
and  characteristics,  while  Section  “E”,  “Student  Expectations,”  asked  them  to  provide 
information  about  their  expectations  before  attending  ACSC.  Together,  Sections  “D”  and 
“E”  made  up  the  input  variables.  Section  “F”,  “Open-Ended  Questions,”  provided 
students  with  an  opportunity  to  give  written  feedback  about  the  program. 

The  questionnaire  was  scored  primarily  using  a  five-point  rating  scale.  In  some 
instances,  students  were  asked  to  place  a  check  mark  next  to  the  most  appropriate  item 
(such  as  in  the  demographic  section).  In  addition  to  the  numerically  scored  items,  the 
open-ended  questions  provided  further  substance  to  the  results  of  the  study  and  helped  to 
interpret  some  of  the  quantitative  data. 

The  initial  questionnaire  was  mailed  on  November  3,  1995.  This  mailing  went  to  all 
members  of  the  sub-population  of  Air  Force  officers  from  the  AY  1994-1995  class  at 
ACSC  (n=395).  The  mailing  consisted  of  a  personalized  cover  letter,  the  questionnaire, 
and  a  self-addressed  stamped  return  envelope.  A  week  after  the  questionnaire  was  mailed, 
a  follow-up  postcard  was  sent  to  all  members  of  the  sub-population.  The  purpose  of  the 
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postcard  was  to  remind  those  who  had  not  sent  a  response  to  return  the  questionnaire. 
Finally,  approximately  three  weeks  after  the  first  questionnaire  was  mailed,  another 
mailing  was  sent  to  those  who  had  not  yet  responded.  The  second  mailing  contained 
another  personalized  cover  letter,  a  replacement  questionnaire,  and  a  self-addressed, 
stamped  return  envelope.  This  three-step  procedure  was  used  to  ensure  that  a  majority  of 
the  population  was  contacted  and  was  designed  to  improve  the  survey  response  rate.  The 
cover  letters,  post  card,  as  well  as  the  questionnaire  are  at  Appendix  A. 

Phot  Study.  The  researcher  visited  ACSC  during  the  week  of  August  14-18, 1995  to 
conduct  a  pilot  study  of  the  survey  instrument.  Because  seventeen  members  of  the  AY 
1994-1995  graduating  class  remained  at  the  College  as  faculty,  they  were  chosen  as 
participants  for  the  pilot  study.  The  researcher  placed  sealed  envelopes  in  staff  mail  boxes 
for  the  chosen  participants,  each  containing  a  questionnaire  and  a  rating  sheet. 

Instructions  directed  the  participants  to  complete  the  questionnaire  as  if  they  had  just 
received  it  in  the  mail  and  to  comment  on  the  instrument’s  structure,  content,  and 
questions  on  the  rating  sheet.  To  facilitate  return  of  the  pilot  survey,  a  box  was  set  up  in 
the  Dean’s  office  and  electronic  mail  reminders  were  sent  to  all  of  the  participants. 

Twelve  of  the  seventeen  (70  percent)  questionnaires  were  returned. 

Because  student  experiences  at  ACSC  did  not  appear  to  vary  significantly,  the 
researcher  was  concerned  that  student  responses  on  the  questionnaire  would  not  vary 
either.  After  the  pilot  study  questionnaires  were  completed  and  returned,  the  researcher 
computed  means  and  standard  deviations  on  every  5th  question  of  the  instrument  in  order 
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to  determine  variability  on  the  responses.  This  method  confirmed  that  student  perceptions 
did  vary  across  the  scale  ranges.  The  researcher  also  was  concerned  about  the  length  of 
the  questionnaire  and  the  participants’  ability  to  understand  and  interpret  the  questions. 
The  pilot  study  revealed  that  it  took  between  15  and  20  minutes  to  complete  the 
instrument  and  that  the  instructions  and  questions  presented  no  problem  for  those  who 
participated  in  the  pilot.  As  a  result  of  the  pilot  study,  the  researcher  was  able  to  refine 
the  instrument  and  meet  the  survey  mailing  deadline  of  November  3,  1995. 

Validity  and  Reliability.  Because  of  the  unique  nature  of  ACSC  and  this  study,  no 
existing  survey  instrument  was  available.  The  pilot  survey  aside,  the  “Student  Perceptions 
of  Program  Effectiveness  Questionnaire”  was  used  for  the  first  time  when  it  was  mailed  as 
part  of  this  study.  However,  every  effort  was  made  to  improve  measurement  quality.  The 
pilot  study  provided  information  about  the  instrument’s  structure  and  organization  as  well 
as  its  content  and  wording.  In  addition,  the  researcher  worked  closely  with  program 
administrators  and  senior  leaders  at  ACSC  to  ensure  that  only  pertinent  questions  were 
addressed.  A  systematic  content  analysis  of  course  syllabi  to  determine  outcomes  also 
enhanced  measurement  quality  of  the  instrument.  To  minimize  measurement  error,  each 
variable  on  the  questionnaire  contained  several  items  rather  than  relying  on  a  single-item 
response  set.  For  example,  the  questionnaire  contains  four  items  (questions  50-53)  to 
measure  the  quality  of  teaching  methods,  an  environmental  variable.  According  to  Light, 
Singer,  and  Willett  (1990),  this  strategy  reduces  measurement  error  while  increasing  the 
reliability  of  the  total  score  (p.  173).  Similarly,  by  combining  multiple  indicators  (both 
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multiple-choice  and  open-ended  questions)  on  the  questionnaire,  measurement  error  was 
minimized  and  the  measurement  quality  improved. 

Data  Analysis 

The  researcher  coded  the  questionnaires  before  they  were  mailed  so  they  could  be 
tracked  for  the  follow-up  mailing.  Although  these  codings  were  associated  with  student 
names  and  addresses,  the  information  was  used  for  tracking  purposes  only.  Respondents 
were  guaranteed  complete  confidentiality  as  the  data  were  reported  in  aggregated  form. 

As  the  questionnaires  arrived  in  the  mail,  the  researcher  entered  the  data  into  a 
computerized  statistical  package.  Computer  software  used  to  analyze  the  data  for  this 
study  was  the  Statistical  Package  for  the  Social  Sciences  (SPSS).  SPSS  is  a  flexible 
statistical  tool  that  can  provide  descriptive,  relational,  and  predictive  information. 

Descriptive  statistics  consisted  of  two  types:  single- variable-descriptive  analyses  and 
relational.  Simple  descriptive  statistics  such  as  means,  standard  deviations,  and 
percentages  were  reported  on  each  item  on  the  questionnaire.  Relational  statistical 
analyses  consisted  of  two  types:  cross  tabulations  and  correlations.  Cross  tabulations 
were  used  to  describe  the  relationship  between  demographics/student  expectations 
(inputs)  and  outcome  variables  and  were  reported  in  percentages.  The  Pearson  product- 
moment  correlation  coefficient  (Pearson  “r”)  was  used  to  report  the  relationship  between 
environmental  variables  and  outcomes.  The  correlation  coefficient  described  the  strength 
and  direction  of  those  relationships. 
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Univariate  correlations  between  variables  are  mainly  associational  and  descriptive.  A 
more  powerful  statistical  tool  for  determining  causation  is  multiple  regression  analysis. 

This  study  used  multiple  regression  analysis  to  identify  the  effect  of  environmental 
variables  at  AC  SC  on  students’  perceptions  of  competence  on  outcomes,  while  controlling 
for  potentially  biasing  input  variables  (demographics  and  student  expectations).  This 
method  allowed  the  researcher  to  make  causal  claims  about  the  perceived  quality  of  the 
environmental  variables  at  the  College  on  student  perceptions  of  their  competence  on 
outcomes.  For  both  the  relational  and  predictive  analyses,  individixal  items  that  made  up 
each  variable  (environment  and  outcomes)  were  added  together  to  create  a  single  value  for 
the  variables.  For  example,  for  Teaching  Methods,  an  environmental  variable,  items  50-53 
were  added  together  to  create  that  variable.  Similarly,  for  Joint  Campaign,  an  outcome 
variable,  items  35-45  were  added  together  to  create  that  variable.  These  “collapsed” 
values  were  reported  in  table  format.  Cross-tabulations  were  used  to  report  the 
relationship  between  inputs  and  outcomes,  while  Pearson  “r”s  were  used  to  report  the 
relationship  between  environment  and  outcomes.  Cross-tabulations  and  Pearson  “r”s 
between  individual  items  that  made  up  each  variable  were  reported  in  Appendices  C  and 
D.  The  researcher  highlighted  patterns  in  the  tables  in  a  short  narrative  that  accompanied 
the  tables. 

Finally,  the  researcher  analyzed  the  open-ended  portions  of  the  questionnaire  through 
a  content  analysis.  The  content  analysis  consisted  of  extracting  students’  key  comments 
and  phrases  and  the  data  were  reported  in  aggregated  form.  Students  were  asked  to 
contribute  information  about  how  AC  SC  could  be  improved.  They  were  also  asked  to 
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comment  on  any  specific  items  on  the  questionnaire  that  they  deemed  appropriate.  These 
data  were  used  to  provide  feedback  to  ACSC  senior  leaders  and  feculty  about  specific 
items  on  the  questionnaire. 


Limitations 

The  first  limitation  in  designing  a  comprehensive  evaluation  revolved  around  the  scope 
of  the  study.  Since  it  was  not  possible  to  study  everything,  the  researcher/evaluator  must 
decide  what  to  study  and,  perhaps  most  importantly,  what  not  to  study.  This  project 
contained  a  number  of  variables  on  which  it  concentrated.  Some  important  questions 
remain:  “Were  these  the  right  variables?”  “Should  others  have  been  included?”  “How 
many  variables  were  enough  to  make  judgments  about  the  perceived  effectiveness  of  this 
program?”  To  minimize  this  concern,  the  evaluator  attempted  to  work  as  closely  as 
possible  with  the  program  stake  holders— senior  ACSC  leaders,  faculty,  and  students. 
These  individuals  will  use  the  results  of  the  study  to  make  changes  in  their  program  and  it 
was  important  for  them  to  be  involved  in  determining  which  variables  were  critical.  In 
addition,  by  involving  principle  stake  holders,  the  researcher  ensured  that  their  needs  were 
met  and  important  variables  were  included.  While  this  dissertation  focused  on  the 
program  as  a  whole,  sacrificing  depth  for  breadth,  this  is  only  one  approach  to  evaluation 
of  ACSC.  In-depth  studies  of  every  variable  included  in  this  study  are  clearly  warranted 
and  should  be  considered  when  future  evaluations  are  planned. 
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Another  limitation  of  the  study  was  a  design  issue.  A  longitudinal  study  consisting  of 
a  pretest  and  a  posttest  would  have  been  ideal.  Such  a  study  would  have  compared 
pretest  and  posttest  scores  to  determine  the  extent  to  which  students  changed  as  a  result 
of  the  program.  It  was  not  possible,  given  the  time  constraints  and  resources  available, 
for  this  study  to  design  and  conduct  a  longitudinal  study  of  this  nature.  While  a 
pretest/posttest  design  would  have  brought  together  all  the  elements  of  classic 
experimental  research  methods,  it  was  not  the  intention  of  this  dissertation  to  provide 
empirical  evidence  in  support  of  a  theory.  Rather,  this  dissertation  was  interested  in 
analyzing  variability  in  students’  perceptions  on  outcome  variables  based  on  information 
available  on  a  number  of  independent  variables.  The  independent  variables  deemed 
important  to  this  study  were  the  input  and  environmental  variables  previously  discussed. 
The  researcher,  based  on  student  perceptions  of  their  competence  in  several  outcome 
variables  and  their  overall  rating  of  program  elements,  wanted  to  determine  which 
elements  of  the  program  best  predicted  students’  perceptions  of  their  competence  on 
outcomes.  According  to  Elazar  Pedhazur  (1982),  in  predictive  research  “any  procedure 
that  meets  the  specific  needs  and  inclinations  of  the  researcher  (economy,  ready 
availability  of  some  variables,  ease  of  obtaining  specific  measurements)  will  do”  (p.  9).  A 
countervailing  principle,  however,  is  the  constraint  that  “when  variables  are  selected  for 
the  purpose  of  optimizing  prediction,  all  one  can  say  is,  given  a  specific  procedure  and 
specific  constraints  placed  by  the  researcher,  which  combinations  of  variables  best  predicts 
the  criterion”  (p.  10).  This  study  was  a  predictive  study  and  was  interested  in  making 
judgments  about  the  program  based  on  available  data.  Future  research  efforts  conducted 
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at  ACSC  should  include  longitudinal  data  using  experimental  designs  which  can  then  be 
compared  to  the  results  of  this  study. 

A  third  limitation  of  the  study  involved  the  issue  of  self  reporting.  Again,  because  of 
time  constraints  and  other  factors,  it  was  not  possible  to  do  a  longitudinal  study  in  which  a 
pretest/post  test  design  could  be  implemented.  In  addition,  because  of  the  ACSC  selection 
process,  a  control  group  could  not  be  identified  and  used.  Therefore,  the  researcher  asked 
the  students  to  rate  themselves  on  their  competence  in  outcomes.  The  credibility  of 
student  self-reports  has  been  often  debated  but  some  evidence  exists  (Pace,  1985)  to 
suggest  that  questionnaire  survey  research  on  which  students  self-report  their  competence 
is  as  valid  as  other  measures.  Pace  analyzed  the  College  Student  Experiences 
Questionnaire  (CSEQ),  a  standardized  questionnaire  designed  for  undergraduates  and 
intended  to  be  filled  out  at  the  end  of  the  school  year.  The  CSEQ  asks  students  to  rate 
their  experiences  on  142  activities,  their  impressions  of  the  college  environment,  and  their 
estimates  of  gains  on  21  goals  and  objectives.  Based  on  test-retest  comparisons  on  the 
CSEQ,  Pace  concluded  that  student  self  reports  were  stable  and  reliable.  In  addition,  he 
compared  self-reported  gains  with  known  achievement  testing  and  concluded  that  “the 
ratings  are  totally  congruent  with  what  we  know  fi-om  achievement  test  scores”  (p.  42). 

A  fourth  limitation  of  this  study  was  that  all  ACSC  students  were  exposed  to  the  same 
teaching  methods  and  program  elements.  In  other  words,  the  environment  for  every 
ACSC  student  was  the  same  with  minor  variations.  Students  were  divided  into  morning 
and  afternoon  seminars  and  they  were  exposed  to  different  faculty  members,  who 
undoubtedly  used  different  approaches.  However,  ACSC  is  a  standardized  course 
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designed  to  meet  the  needs  of  a  majority  of  its  students.  Basing  this  study  on  student 
perceptions  was  one  way  in  which  the  lack  of  variation  could  be  addressed.  Student 
perceptions  do  vary,  according  to  the  pilot  study,  and  revealed  strengths  and  weaknesses 
in  the  program. 

Although  this  study  did  not  attempt  to  address  every  issue  at  ACSC,  it  will  serve  as  a 
building  block  upon  which  fixture  studies  can  be  built.  There  are  several  implications  for 
future  study.  First,  a  longitudinal  study  is  clearly  warranted.  Such  a  study  would  possibly 
administer  student  tests  at  the  beginning  of  the  ten-month  period  and  follow  through  with 
a  post  test  at  the  end.  Second,  research  needs  to  be  done  on  the  selection  process. 
Differences  between  those  selected  for  ACSC  and  those  not  selected  should  be  measured 
to  determine  the  value  of  the  program.  In  this  case,  the  non-selectees  would  serve  as  a 
control  group  which  would  bring  more  validity  to  any  causal  claims  made  about  student 
achievement.  Finally,  in-depth  studies  could  be  done  on  any  number  of  the  variables.  For 
example,  the  technology  element  described  as  an  environmental  variable  in  this  study, 
deserves  more  attention  than  this  study  could  give  to  it.  The  same  could  be  said  for  any  of 
the  variables  mentioned  in  the  study. 

In  conclusion,  the  limitations  and  ideas  for  future  research  notwithstanding,  this  study 
served  to  provide  information  about  the  overall  perceived  quality  of  ACSC’s  program 
activities  and  predict  which  elements  contributed  to  students’  perceptions  of  competence 
on  outcomes.  It  will  hopefully  solicit  discussions  and  conversations  among  faculty, 
students,  and  staff  and  serve  to  bring  about  program  change  and/or  improvements. 
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Finally,  the  researcher  hopes  that  this  dissertation  will  be  a  building  block  upon  which 
future  research  can  be  built.  By  providing  useful  information  to  stake  holders  and  by 
developing  a  comprehensive  evaluation  methodology,  this  study  will  provide  long-term 
benefits  for  Air  Command  and  Staff  College  and  to  those  who  care  about  its  central  role  in 
professional  military  education. 
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CHAPTER  FOUR 

RESULTS  AND  FINDINGS 

This  chapter  covers  the  results  and  findings  of  the  data  analysis.  To  best  present  the 
results  of  this  study,  the  analysis  is  divided  into  three  separate  categories:  descriptive, 
relational,  and  predictive/causal.  Each  category  is  covered  separately  and  data  analysis 
results  correspond  to  the  research  questions  posed  in  chapters  one  and  three  of  this 
dissertation.  For  the  reader’s  convenience,  the  research  questions  are  repeated  at  the 
beginning  of  each  section.  Following  the  research  questions,  results  are  summarized  in 
tables  and  patterns  are  highlighted  in  a  brief  narrative.  In  each  section,  items  fi’om  the 
questionnaire  that  correspond  to  the  analysis  are  indicated.  After  the  quantitative  data  are 
presented,  the  qualitative,  open-ended  questions  (89  and  90  on  the  questionnaire)  are 
summarized  in  a  final  section. 

Before  turning  to  the  in-depth  analysis  of  the  data,  it  is  important  to  understand  to 
what  extent  the  respondents  mirror  the  population  from  which  they  came.  A  discussion  on 
representativeness  will  give  the  reader  confidence  that  those  who  responded  to  the 
questionnaire  were  characteristic  of  the  study  population,  thereby  making  generalizations 
from  the  data  possible. 


Representativeness 

The  study  group  chosen  for  this  study  consisted  of  all  395  Air  Force  officer  graduates 
from  the  ACSC  Academic  Year  (AY)  1994-1995  class.  Of  the  395  questionnaires  sent  to 
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alumni,  228  were  returned  (58  percent)  and  107  came  back  as  “undeliverable  as 
addressed”  mail  (27  percent).  Seventy  percent  of  the  undeliverable  questionnaires  (75) 
were  returned  from  the  same  address—Air  Command  and  Staff  College.  While  a  few 
graduates  remained  at  the  College  as  faculty  and  staff,  the  large  number  of  undeliverable 
mail  to  ACSC  reflects  a  time  lag  in  the  assignment  process  rather  than  a  qualitative 
difference  in  recipients.  Graduates  who,  for  one  reason  or  another,  received  late 
assignments  or  attended  other  courses  after  ACSC,  would  possibly  be  listed  incorrectly  in 
the  personnel  system.  There  is  no  reason  to  believe  that  graduates  who  could  not  be 
reached  because  of  bad  addresses  are  significantly  different  than  those  who  responded  to 
the  survey. 

Although  the  response  rate  is  important  because  of  the  possibility  of  nonresponse  error, 
of  greater  importance  is  the  representativeness  of  the  returned  surveys.  Nonresponse 
error  occurs  when  a  significant  number  of  people  do  not  respond  to  the  survey  and  they 
are  different  from  respondents  in  ways  that  are  important  to  the  study  (Salant  and  Dillman, 
1994,  p.  20).  Representativeness  refers  to  the  extent  that  the  characteristics  of  people 
who  respond  to  the  survey  are  similar  to  the  characteristics  of  the  population.  This  is 
important  because  in  order  to  make  generalizations  based  on  data  provided  by 
respondents,  the  researcher  must  have  confidence  that  those  who  participated  in  the  study 
represented  the  study  group  as  a  whole. 

The  data  comparison  on  representativeness  of  respondents  to  study  group  is 
summarized  in  Table  1 .  Data  on  the  study  group  were  provided  by  ACSC  and  were  listed 
in  their  .4796  Curriculum  Plan  (March  1995, 14-15).  Totals  and  percentages  of 
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respondents  in  each  category  were  a  result  of  compiling  demographic  data  from  returned 
questionnaires.  The  figures  reported  do  not  include  responses  checked  in  the  “other” 
category.  The  number  of  respondents  checking  “other”  in  every  demographic  variable 
were  too  small  to  be  of  any  use  in  the  comparisons.  Some  respondents  left  demographic 
data  blank  so  the  figures  reported  represent  only  valid  responses. 


Table  1 


Representativeness  of  Respondents  to  Study  Group 


STUDY  GROUP  (395) 

RESPONDENTS  (228) 

Age  (yrs.) 

37.1 

35- under  24.5% 

36- 39  52.3  % 

40-over  23.2% 

Gender 

Male 

Female 

318  (81%) 

77  (19%) 

179  (81.4%) 

41  (18.6%) 

Race 

Caucasian 

African-American 

Other 

339  (86%) 

36  (  9%) 

20  (  5%) 

181  (83%) 

17  (  7.8%) 

24  (8.2%) 

Career  Field 

Rated  (pilots,  navigators) 
Non-rated  (all  others) 

128  (32%) 

267  (68%) 

70  (32%) 

135  (66%) 

Education 

Bachelor 

Master 

Doctorate 

39  (10%) 

339  (86%) 

8(  2%) 

4(  1.8%) 

207  (90.8%) 

13  (  5.9%) 

A  comparison  of  demographic  data  between  the  study  group  and  the  respondents 
reveal  that  those  who  responded  to  the  study  were  very  similar  to  the  population  from 
which  they  came.  The  mean  age  of  Air  Force  oflScer  graduates  was  reported  by  ACSC  as 


76 


37  years  old.  The  majority  of  respondents  (52.3  percent)  to  the  survey  fell  in  the  36-39 
year  old  category  on  the  questionnaire.  Comparisons  of  gender  and  race  between  the  two 
groups  show  extreme  similarities.  Males  in  the  study  group  made  up  81  percent  of  the 
population  while  females  consisted  of  19  percent.  Respondents’  totals  were  almost 
identical  to  the  study  group  figures  with  males  consisting  of  81.4  percent  and  females, 

1 8.6  percent.  Figures  for  race  were  not  as  close  in  comparison  as  gender  but  are  still 
representative.  ACSC  reported  that  86  percent  of  Air  Force  students  for  AY  1994-1995 
were  Caucasian,  nine  percent  were  Afi'ican-American,  and  five  percent  were  listed  as 
“other  minorities.”  Data  compiled  fi’om  the  questionnaire  on  race  show  that  83  percent  of 
respondents  were  Caucasian,  while  7.8  percent  were  African-American,  and  8.2  percent 
fell  in  the  “other  minorities”  category.  These  figures  show  that  a  lower  percentage  of 
Afiican- American  students  responded  to  the  survey  than  occurred  in  the  population. 
Conversely,  a  higher  percentage  of  respondents  listed  themselves  as  “other  minorities”  on 
the  questionnaire  than  those  in  the  study  group.  This  may  have  occurred  because  the  way 
ACSC  defined  “other  minorities”  and  the  way  they  were  defined  on  the  questionnaire 
could  have  been  different.  Comparatively,  combined  percentages  of  aU  minorities 
between  the  study  group  and  respondents  were  very  close  overall  (14  vs.  16  percent). 

Although  gender  and  race  were  the  most  important  categories  in  determining 
representativeness,  other  interesting  demographic  data  emerged  fi'om  the  comparison.  The 
data  show  almost  identical  statistics  between  the  study  group  and  the  respondents  in  the 
career  field  category.  For  both  the  study  group  and  respondents,  thirty-two  percent  (32 
%)  of  the  study  group  were  rated  officers  (pilots  and  navigators),  while  the  difference 


77 


between  the  two  groups  in  the  non-rated  (all  others)  category  was  only  two  percent  (2  %). 
Statistics  in  the  education  category  revealed  an  interesting  comparison.  While  Air  Force 
officers  who  reach  the  rank  of  Major  are  expected  to  have  earned  Master  Degrees,  figures 
provided  by  ACSC  on  education  revealed  that  ten  percent  (10  %)  of  Air  Force  officer 
graduates  had  Bachelor  Degrees  only.  Eighty-six  percent  (86  %)  had  Master  Degrees  and 
only  two  percent  (2  %)  indicated  that  they  had  earned  Doctorates.  Conversely,  fewer 
respondents  (1.8  percent)  had  only  Bachelor  Degrees,  while  almost  ninety-one  percent 
(90.8%)  had  Master  Degrees  and  nearly  six  percent  (5.9%)  had  earned  Doctorates. 

Overall,  the  respondents  were  better  educated  than  the  study  group.  The  larger  niunber  of 
respondents  with  Doctorates  may  be  partially  explained  by  the  feet  that  they  may  be  more 
appreciative  of  the  research  process  and  do  not  hesitate  participating.  While  the 
questionnaire  asked  participants  to  list  their  Master  Degree  concentration,  ACSC  does  not 
report  that  information,  therefore,  a  one-on-one  comparison  was  not  possible.  However, 
a  comparison  of  the  educational  level  shows  a  high  degree  of  representativeness  in  the 
education  category.  ACSC  also  does  not  report  Air  Force  officer  only  commissioning 
sources.  Rather,  they  group  all  military  branches  (Army,  Navy,  Air  Force,  Marines,  etc.) 
together  and  report  commissioning  source  of  all  officers.  Therefore,  a  one-on-one 
comparison  of  commissioning  source  was  not  possible  either. 

This  comparison  between  study  group  demographics  and  respondents  reveals  close 
similarities  in  the  two  groups.  Across  all  categories  examined,  those  who  responded  to 
the  questionnaire  were  representative  of  the  study  group  population.  As  a  result,  the 
possibility  of  nonresponse  error  was  lessened.  In  addition,  the  researcher  and  those 
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interested  in  this  research  can  have  greater  confidence  in  the  findings  as  representativeness 
has  been  established,  making  generalizations  possible. 

Descriptive  Analysis 

Specific  descriptive  research  questions  are  as  follows:  “What  were  the  demographic 
factors  and  student  expectations  of  ACSC  graduates  in  academic  year  1994-1995?” 

“How  did  students  rate  the  quality  of  the  teaching  methods  and  program  activities  at  Air 
Command  and  Staff  College?”  and  “How  did  students  rate  their  competency  on  outcome 
measures  at  the  time  they  completed  ACSC?”  The  first  question  concerned  demographic 
data  and  student  expectations.  Demographic  data  were  compiled  fi’om  student  responses 
on  the  questionnaire,  items  82-87,  and  student  expectations  fi'om  question  88.  To  answer 
the  second  question  concerning  the  quality  of  ACSC’s  teaching  methods  and  program 
activities,  the  researcher  compiled  data  fi’om  items  50-81  on  the  questionnaire.  Similarly, 
to  answer  question  three  concerning  student  perceptions  on  their  competency  on  outcome 
measures,  items  5-49  were  used. 

Demographic  Factors  and  Student  Expectations  (Input  Variables') 

A  complete  breakdown  of  demographic  data  fi'om  the  questionnaire  is  summarized  in 
Table  2.  Although  much  of  this  information  was  already  presented  in  the  earlier 
discussion  on  representativeness,  the  data  covered  in  Table  2  are  more  detailed  and 
correspond  to  Section  “D”  on  the  questionnaire. 


Table  2: 
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Item 

Percentages 

Age  (yrs.) 

35- under  24.5% 

36- 39  52.3% 
40-over  23.2% 

Gender 

Male 

Female 

179  (81.4%) 

41  (18.6) 

Race 

African-American 

Asian 

Hispanic 

Native- American 
White 

17(7.8%) 

4(1.8%) 

10  (4.6%) 
4(1.8%) 

181  (83%) 

Career  Field 

Operations 

Mission  Support 

Logistics 

Medical/Legal 

82  (37.3%) 

75  (34.1%) 
42(19.1%) 

6  (2.7%) 

Master  Degree 
Humanities 

Social  Sciences 

Math  and  Science 

Engineering 

Medical/Legal 

11  (5.0%) 

108  (49.3%) 

52  (23.7%) 

34  (15.5%) 

6  (3.2%) 

Commissioning  Source 
ROTC 

USAFA 

OTS 

91  (42.2%) 

44  (19.9%) 

78  (35.3%) 

The  average  ACSC  graduate  from  AY  1994-1995  was  a  white  male  between  the  ages  of 
36-39.  A  higher  percentage  of  respondents  worked  in  the  Operations  Career  Field  (pilots, 
navigators,  space  systems,  missileers,  air  weapons  and  air  traffic  controllers)  than  in  any 
other  career  field.  A  breakout  of  career  fields  is  provided  to  show  specific  job-related 
duties  that  might  have  an  effect  on  outcomes.  The  information  on  the  table,  however,  may 
be  somewhat  misleading.  While  it  appears  as  if  the  greatest  percentage  of  survey 
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respondents  were  “operators,”  in  reality,  most  ACSC  graduates  were  non-operational. 
This  fact  is  calculated  by  adding  the  other  career  field  totals  together  to  determine  the 
percentage  of  those  who  work  outside  the  Operations  career  field  (55.9  %).  One  can 
conclude  from  these  data  that  the  majority  of  respondents  did  not  directly  participate  in 
the  war  making  capability  of  the  Air  Force.  Rather,  they  were  involved  in  duties  that 
supported  that  mission.  Educationally,  more  graduates  were  commissioned  through  the 
Reserve  Officer  Training  Corps  (ROTC)  than  either  Officer  Training  School  (OTS)  or  the 
Air  Force  Academy.  Finally,  when  they  sought  advanced  degrees,  more  chose  to  major  in 
the  Social  Sciences  than  all  other  majors  combined. 

Student  expectations  refer  to  Section  “E”  on  the  questionnaire  and  provide  information 
about  what  students  expected  ACSC  to  do  for  them  before  they  attended  the  College. 
Students  were  asked  to  respond  to  the  question:  “When  I  found  out  that  I  was  selected  to 
attend  ACSC,  1  ejqjected  that:. . . .”  There  were  nine  possible  expectations  fi'om  which 
students  could  choose.  The  list  consisted  of  positive  and  negative  statements  that  were 
chosen  based  on  anecdotal  information  fi'om  students  (past  and  present),  faculty,  and  staff 
Choices  covered  both  personal  and  professional  expectations.  Although  the  list  was  not 
meant  to  be  exhaustive,  it  provided  students  with  choices  that,  in  the  researcher’s  opinion, 
formed  the  core  of  possibilities.  Student  expectations  are  summarized  in  Table  3. 


Table  3: 

Student  Expectations  Summary 
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Student  Expectations 

(in  percentages) 

When  I  found  out  that  I  was  selected  to  attend  ACSC,  I  expected  that: 

1.  ACSC  would  improve  my  chances  for  future  promotions. 

86.0 

2.  ACSC’s  curriculum  would  make  me  a  better  Air  Force  officer. 

85.5 

3.  I  would  meet  other  people  and  learn  about  other  career  fields. 

88.3 

4.  I  would  improve  my  golf  game. 

16.2 

5.  I  would  get  to  spend  more  time  with  my  &mily. 

46.8 

6.  I  would  spend  a  great  deal  of  time  socializing. 

18.9 

7.  I  would  learn  very  little  ^^ile  a  student  at  ACSC. 

2.7 

8.  I  would  be  academically  challenged  wiiile  a  student  at  ACSC. 

64.4 

9.  Other 

11.7 

This  summary  of  student  expectations  reveals  that  ACSC  graduates  expected  the 
College  to  be  a  professional  school  in  which  they  would  be  academically  challenged.  They 
appeared  to  understand  the  value  of  networking  skills  as  is  evidenced  by  their  responses  to 
expectation  three  (88.3  %).  Similarly,  they  seemed  to  realize  the  value  of  ACSC  to  their 
career  progression  based  on  their  responses  to  questions  one  and  two  (86  %  and  85.5  % 
respectively).  On  the  personal  level,  a  few  responded  that  they  expected  to  spend  a  great 
deal  of  time  socializing  and/or  playing  golf.  Of  greater  import,  on  a  personal  level,  was 
the  expectation  that  graduates  could  spend  more  time  with  their  femily  which  was 
reflected  in  their  responses  to  expectation  five  (46.8  %),  Expectation  nine  provided 
respondents  with  an  opportunity  to  add  to  the  list.  Of  the  1 1 .7  percent  who  checked  the 
“Other”  block,  the  majority  of  comments  fell  in  the  professional  arena.  Many  expected 
that  ACSC  would  give  them  skills  that  would  make  them  better  commanders,  while  others 
saw  ACSC  as  a  place  to  learn  about  current  Air  Force  issues  and  to  set  up  professional 
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networks.  Personal  comments  from  the  “Other”  block  were  few  but  revolved  around 
student  expectations  that  they  would  have  time  while  at  ACSC  to  complete  their  Master 
Degree.  One  student  actually  attended  classes  at  Auburn  University  while  a  student  at 
ACSC. 

The  descriptive  summary  compiled  from  demographic  and  student  expectations  data 
reveals  that  those  who  responded  to  the  survey  were  representative  of  the  study  group.  It 
also  reveals  that  they  were  highly  interested  in  furthering  their  professional  goals.  While 
they  were  concerned  about  their  careers,  they  were,  by  and  large,  family  oriented.  These 
data  are  important  because  it  gives  the  uninitiated  reader  information  about  this  unique 
population  and  their  motivations. 

Program  Activities  and  Teaching  Methods  (Environmental  Variables! 

The  environmental  variables  consisted  of  six  categories.  They  were: 

•  Teaching  Methods 

•  Technology 

•  Curriculum 

•  Research 

•  Faculty 

•  Grading 

Descriptive  data  on  these  categories  were  compiled  from  student  responses  to  items  50-81 
on  the  questionnaire.  Students  were  asked:  “How  would  you  rate  the  quality  of  the 
following  program  elements  at  Air  Command  and  Staff  College?”  A  five-point  rating 
scale  was  used  to  rate  their  perceptions  about  the  quality  of  teaching  methods  and 
program  activities  at  the  College.  Each  category  is  summarized  separately  by  grouping 
appropriate  questions  together  to  form  the  six  categories  mentioned  above. 
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Teaching  Methods.  Data  compiled  on  teaching  methods  (items  50-53  on  the 
questionnaire)  are  summarized  in  Table  4.  The  table  shows  that,  on  the  average,  students 
rated  the  quality  of  lectures  by  guest  lecturers  higher  than  other  teaching  methods  used  at 
ACSC.  An  important  observation  is  that,  of  all  the  teaching  methods,  lectures  by 
instructors  were  rated  the  lowest.  Yet,  one-third  of  respondents  rated  lectures  by 
instructors  as  either  “excellent”  and  “outstanding.”  By  comparison,  over  two-thirds  of 
respondents  perceived  guest  lecturers  as  either  “excellent”  or  “outstanding.”  Students 
seemed  to  perceive  that  both  informal  lecture  seminars  and  colloquia  style  seminars  were 
similar  in  quality.  The  above  observations  notwithstanding,  the  overall  mean  of  3.3  was 
relatively  weak,  showing  a  lack  of  enthusiasm  for  teaching  methods  used  at  the  CoUege. 


Table  4; 


Teaching  Methods  listed  by  item 


Teaching  Methods 

(in  percentages) 

Item 

Unsat 

(1) 

Marg 

(2) 

Sat 

(3) 

Excel 

(4) 

Out 

(5) 

Mean 

SD 

Lectures  by  instructors 

2.2 

14.9 

49.1 

31.1 

2.6 

3.17 

.79 

Lectures  by  guest  lecturers 

1.3 

4.8 

26.0 

55.5 

12.3 

3.73 

.79 

Informal  lecture  seminars 

2.7 

8.9 

47.1 

36.0 

5.3 

3.32 

.82 

Colloquia  style  seminars 

1.9 

5.2 

53.1 

35.7 

4.2 

3.35 

.73 

Teaching  Methods  Overall  Mean  =  3.32 

Technology.  The  technology  category  consisted  of  7  items  (54-58,  74,  and  81).  Table 
5  summarizes  data  on  items  54-58.  Summary  data  on  item  #74  from  the  questionnaire 
appear  in  table  6.  Table  7  lists  summary  information  on  question  #8 1 . 

All  ACSC  students  were  issued  a  portable  computer  with  a  486  microprocessor.  The 
portables  had  CD-ROM  capability  and  could  be  networked  together  so  that  students  could 
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send  electronic  mail  to  their  classmates,  feculty,  or  staff.  The  AY  1994-1995  class  was 
only  the  second  class  to  have  this  technology  so  readily  available  to  them.  While  many  Air 
Force  officers  have  used  computers  in  their  jobs,  most  of  them  have  not  been  exposed  to 
the  latest  innovations  in  computer  technology.  ACSC  attempted  to  provide  some 
instruction  to  those  who  were  not  femiliar  with  the  technology.  Students  were  expected 
to  use  their  computers  daily  as  all  daily  schedules  were  released  through  the  computer. 
Some  instructors  used  electronic  testing  and  many  placed  readings  on  the  network. 
Toolbook©  is  an  interactive  software  that  uses  graphics,  pictures,  and  quick-time  movies 
to  enliven  text.  Toolbook©  was  used  extensively  at  ACSC  with  many  students  creating 
Toolbook©  presentations  as  their  research  projects. 

Table  5: 


Technology 

(in  percentages) 

Item 

Unsat 

(1) 

Marg 

(2) 

Excel 

(4) 

Out 

(5) 

Mean 

SD 

Use  of  computers  for  daily 
schedules 

.4 

5.7 

11.5 

38.3 

44.1 

4.20 

.89 

.9 

7.1 

14.6 

39.8 

37.6 

4.06 

.94 

Toolbook©  readings 

11.0 

14.9 

29.8 

25.0 

19.3 

3.27 

1.24 

Computerized  readings 

39.0 

34.6 

14.9 

7.9 

3.5 

2.02 

1.08 

Computer  Instruction 

12.7 

30.3 

39.9 

12.7 

4.4 

2.66 

1,00 

Technology  overall  mean  =  3.23 

Table  5  shows  how  students  rated  technology  items.  By  comparing  means,  one  can  see 
that  students  rated  highly  the  use  of  computers  for  daily  schedules  and  for  testing  but 
foimd  computerized  readings  and  computer  instruction  less  than  “satisfectory.” 
Toolbook©  readings  were  rated  “satisfactory”  but  respondents  were  much  less  impressed 
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with  them  than  they  were  with  their  use  for  daily  schedules  and  testing.  In  addition,  by 
glancing  at  standard  deviations  (the  last  column  in  the  table)  one  can  compare  the  degree 
of  variability  in  students’  responses  between  the  different  items.  According  to  these 
figures,  Toolbook©  readings  appealed  to  some  students  but  not  others.  However,  forty- 
five  percent  of  respondents  rated  Toolbook©  as  either  “excellent”  or  “outstanding.” 

While  this  information  does  not  reveal  why  students  agreed  less  about  those  items,  it  does 
indicate  that  the  use  of  Toolbook©  programs  and  computerized  readings  at  ACSC  may  be 
cause  for  concern  and  prompt  stake  holders  to  investigate  fiirther. 

Every  graduate  of  ACSC  received  a  CD-ROM  upon  graduation.  This  CD-ROM 
contained  a  variety  of  readings,  Toolbook©  programs,  and  student  research  projects.  It 
was  a  compact  way  to  provide  students  with  materials  used  at  the  College.  Since  the  use 
of  this  type  of  technology  is  relatively  new  at  ACSC,  stake  holders  were  interested  in 
knowing  the  extent  to  which  graduates  found  the  CD-ROM  to  be  useful.  Table  6  and  7 
summarize  data  which  answer  this  question  (item  74  and  81  on  questionnaire). 


Table  6: 

_ 

To  what  extent  have  you  found  the  ACSC  curriculum  CD-ROM  to  be  useful? 


Not  at 

Very 

To  Some 

To  Considerable 

To  Great 

all(l) 

little  (2) 

Extent  (3) 

Extent  (4) 

Extent  (5) 

37.6  % 

19.9  % 

24.4  % 

14.5  % 

3.6  % 

Mean  =  2.27  SD=1.21 

The  data  indicated  on  table  8  show  that  the  average  student  at  ACSC  found  very  little  use 
for  the  CD-ROM,  as  evidenced  by  the  mean  of  2.27.  The  mean,  however,  is  misleading 
because  of  the  large  standard  deviation  of  1.21.  This  shows  that  students  did  not  agree 
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on  the  extent  to  which  the  CD-ROM  was  useful.  However,  by  combining  percentages, 
one  can  see  that  while  57.5  percent  said  they  found  the  CD-ROM  to  be  of  no  and  very 
little  use,  only  42.5  percent  perceived  it  to  be  of  at  least  some  use.  These  figures  indicate 
that  students,  more  often  than  not,  did  not  perceive  the  CD-ROM  as  being  useful  to  them. 
Table  7  corroborates  these  data,  indicating  that  a  large  percentage  of  graduates  claimed 
they  did  not  use  the  CD-ROM  (54.4%).  Written  comments  by  some  respondents 
revealed  that  they  would  use  the  CD-ROM  if  they  had  the  capability.  Many  lamented  the 
fact  that  their  computers  at  work  did  not  have  CD-ROMs  and  that  their  own  personal 
computers  were  also  lacking.  Regardless,  it  appears  as  if  the  CD-ROM  issued  to  students 
upon  graduation  had  limited  utility. 


Table  7 
CD-ROM  Use 


If  you  use  the  CD-ROM,  where  do  you  use  it? 

Home 

Work 

Both 

Don’t  Use 
It 

No 

Response 

17.1% 

19.7  % 

6.6  % 

54.4  % 

2.2  % 

Curriculum.  The  curriculum  category  consisted  of  7  items  (59-62 , 67,  75,  and  76)  on 
the  questionnaire.  Results  of  the  data  analysis  on  items  59-62  are  summarized  in  table  8. 
Item  67  information  appears  in  table  9.  Questions  75  and  76  are  covered  by  tables  10  and 
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Table  8: 

Curriculum  listed  by  item 


Curriculum 

(in  percentages) 

Item 

Unsat 

(1) 

Marg 

(2) 

Sat 

(3) 

Excel 

(4) 

Out 

(5) 

Mean 

SD 

Assigned  Books 

.9 

3.1 

22.4 

46.9 

26.8 

3.96 

.83 

Curriculum  Flow 

4.4 

11.5 

51.8 

4.0 

3.16 

.84 

Systems  Approach 

1.3 

10.1 

31.7 

44.9 

11.9 

3.56 

.88 

Balance  between 
Academics  &  Social 

6.6 

21.1 

37.0 

28.6 

6.6 

3.08 

1.01 

Curriculum  Overall  Mean  =  3.42 


The  ciuriculiim  overall  average  (mean)  fells  halfway  between  “satisfectory”  and 
“excellent”  indicating  that  graduates  perceived  the  ACSC  curriculum  on  the  items 
measured  to  be  of  moderate  quality.  One  hern  (assigned  books)  stands  out  as  being  the 
most  highly  rated.  Students  received  over  100  books  which  they  were  allowed  to  keep 
after  graduation.  They  appear  to  be  very  satisfied  with  the  quality  of  these  books  as 
evidenced  by  the  average  rating  of  3,96  with  over  a  quarter  of  respondents  rating  the 
assigned  books  as  “outstanding.”  The  two  lowest  rated  items  fi’om  table  8  were 
curriculum  flow  and  the  balance  between  academics  &  social.  Yet,  eighty  percent  (80  %) 
of  respondents  rated  the  curriculum  flow  as  “satisfectory”  or  “excellent,”  and  sfacty-six 
percent  (66  %)  perceived  the  balance  between  academics  and  social  as  being  “satisfectory” 
or  “excellent.”  Even  though  the  balance  between  academics  &  social  received  the  lowest 
rating  on  the  average,  the  standard  deviation  of  1.01  indicates  that  students  were  not  in 
agreement  about  this  item. 

Question  67  revolved  aroimd  whether  or  not  graduates  perceived  ACSC  to  be  a 
graduate-level  course.  Data  compiled  and  summarized  in  Table  9  reveal  that  nearly  sixty- 
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one  percent  (60.8  %)  thought  that  ACSC  was  a  graduate-level  course  to  a  “considerable 
extent”  and  to  a  “great  extent.”  However,  the  standard  deviation  of  1 .07  also  indicates 
that  student  responses  on  this  item  varied  widely. 


Table  9: 


To  what  extent  do  yon  think  ACSC  is  a  graduate-level  program' 

9 

Not  at 

Very 

To  Some 

To  Considerable 

To  Great 

all  (1) 

little  (2) 

Extent  (3) 

Extent  (4) 

Extent  (5) 

4.4  % 

8.8  % 

26.0  % 

37.0  % 

23.8  % 

Mean  =  3.67  SD=1.07 

When  asked  about  the  amount  of  assigned  reading  (question  #75)  59.2%  said  there 
was  too  much.  Not  surprisingly,  no  one  responded  in  the  “not  enough”  category.  Data 
from  this  question  are  summarized  in  table  10. 


Table  10: 


1  What  did  you  think  about  the  amount  of  assigned  I 

reading? 

Too  Much 

Not  Enough 

About  Right 

No  Response 

59.2  % 

0% 

38.2  % 

2.6  % 

Written  comments  corroborate  the  figures  with  many  respondents  writing  that  the  number 
of  pages  assigned  detracted  from  their  learning  experiences. 

While  graduates  perceived  that  the  amount  of  reading  was  too  much,  they  felt  more 
positively  toward  the  length  of  courses.  Table  1 1  lists  their  responses  to  the  question  on 
the  length  of  courses. 
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Table  11: 


What  did  you  think  about  the  length  of  the  courses? 

Too  Long 

Too  short 

About  Right 

No  Response 

18.0% 

3.1  % 

75% 

3.9  % 

Although  the  table  shows  that  most  students  perceived  the  overall  length  of  courses  to  be 
“about  right,”  their  written  comments  provided  more  detailed  information.  When  given  an 
opportunity  to  express  their  views,  some  students  wrote  that  although  they  thought  most 
courses  were  long  enough,  one  course,  the  Command  and  Leadership  Course,  should  be 
expanded  and  given  more  time  in  the  curriculum.  Still  others  wrote  that  not  enough 
attention  was  paid  to  space  applications.  These  observations  shed  additional  light  on  the 
data  presented  in  table  1 1 . 

Research.  Research  consisted  of  four  items  (63,  68, 69,  77)  dealing  with  group 
research  projects,  training,  support,  and  research  time.  Data  compiled  on  group  research 
are  summarized  in  table  12,  while  information  about  training,  and  support  appears  in  tables 
13  and  14  respectively.  Research  time  is  reported  in  table  15. 

Descriptive  data  analysis  on  group  research  projects  show  that  graduates  perceived  it 
to  be  one  of  the  lowest  rated  categories.  This  data  are  condensed  in  table  12.  The  overall 
mean  of  2.7  indicates  an  average  rating  that  is  not  quite  “satisfactory.”  A  standard 
deviation  of  1 .4,  however,  shows  that  student  ratings  varied  considerably  across  the  range 
of  scores.  The  combination  of  a  low  mean  and  considerable  variability  suggests  that  the 
group  research  component  of  the  College  may  be  cause  for  concern. 
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Table  12: 


Group  Research 

(in  percentages) 

Item 

Unsat 

(1) 

Marg 

(2) 

■3H 

Excel 

(4) 

Out 

(5) 

Mean 

SD 

Group  Research  Projects 

21.1 

23.2 

31.6 

17.1 

6.6 

2.70 

1.40 

Group  Research  Overall  Mean  =  2.70 


Questions  68  and  69  on  the  questionnaire  dealt  with  research  traming  and  support. 
Students  may  arrive  at  the  College  without  the  necessary  training  to  accomplish  a  quality 
research  project  and  their  reliance  on  feculty  and  staif  could  be  heavy  at  times.  Senior 
leaders  at  ACSC  were  interested  in  how  students  perceived  this  element  of  their  education 
while  a  student  in  the  program.  Tables  13  and  14  consolidate  data  on  these  topics. 


Table  13: 


To  what  extent  were  you  given  adequate  training  on  how  to  do  research? 

Not  at 

Very 

To  Some 

To  Considerable 

To  Great 

all  (1) 

little  (2) 

Extent  (3) 

Extent  (4) 

Extent  (5) 

16.8  % 

42.9  % 

35.8  % 

3.5  % 

.9% 

Mean  =  2.29  SD  =  .82 


Table  14: 

To  what  extent  did  you  receive  adequate  support  with  your  research  project? 


Not  at 

Very 

To  Some 

To  Considerable 

To  Great 

all  (1) 

little  (2) 

Extent  (3) 

Extent  (4) 

Extent  (5) 

15.9  % 

27.4  % 

34.1  % 

15.0  % 

7.5  % 

Mean  =  2.71  SD=1.13 
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Data  from  these  tables  reveal  that  students  perceived  the  quality  of  training  and  support 
for  their  research  somewhat  lacking.  Almost  sixty  percent  (60  %)  of  graduates  replied 
that  they  received  no  or  very  little  training  on  how  to  do  research.  Respondents’  ratings 
of  research  support  was  more  positive  with  forty-three  percent  (43  %)  responding  in  the 
“not  at  all”  or  “very  little”  category  while  forty-nine  percent  (49  %)  replied  that  they 
received  “some”  or  “considerable”  support  on  their  research  projects.  Note,  however, 
that  the  standard  deviation  for  research  support  was  a  1.13,  indicating  a  wider  range  of 
scores  and  showing  that  students  were  not  in  agreement  about  this  item.  Written 
comments  in  open-ended  questions  also  show  dissatisfection  with  the  research  component 
of  the  program.  Students  wrote  that  they  received  very  little  help  with  their  research  and 
that  grading  on  research  projects  was  not  standardized.  This  may  indicate  a  problem  area 
and  warrants  further  investigation. 

A  final  topic  in  the  research  category  was  that  of  time  allotted  to  do  research.  Students 
were  asked:  “What  did  you  think  about  the  amount  of  time  allotted  to  research?’  Their 
responses  are  summarized  in  table  15. 


Table  15: 
Research  Time 


1  What  did  you  think  about  the  amount  of  time  allotted  I 

to  research? 

Too  Much 

Not  Enough 

About  Right 

No  Response 

4.8  % 

56.1% 

36.8  % 

2.2  % 

The  percentage  of  students  who  thought  there  was  not  enough  time  allotted  to  research 
far  exceeded  those  who  checked  aU  other  areas  combined.  Clearly,  time  allotted  to 
research  may  be  another  area  of  concern  and  should  be  reviewed. 
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Faculty.  It  was  not  the  intent  of  this  study  to  single  out  any  one  individual  or 
department  by  asking  students  to  rate  faculty.  Of  particular  concern  was  the  balance 
between  civilian  and  military  faculty  members  and  the  number  of  feculty  members  who 
held  Ph.D.s.  In  keeping  with  their  desire  to  be  comparable  to  graduate  study,  ACSC 
senior  leaders  wanted  to  know  graduate  perceptions  on  these  items.  The  results  of  these 
items  are  made  even  more  significant  by  the  fact  that  ACSC  recently  asked  for  and  was 
allocated  money  to  fund  12  individuals  for  Ph.D.  study.  In  addition,  several  military  slots 
are  scheduled  to  be  converted  to  civilian  positions.  The  feculty  currently  has  very  few 
civilian  instructors  and  even  fewer  Ph.D.s  but  the  trend  is  to  hire  more  of  both. 

Table  16  compares  student  perceptions  of  the  quality  of  military  vs.  civilian  faculty 
(items  64  and  65  on  the  questionnaire).  It  appears  as  if  civilian  feculty  members  were 
rated  of  higher  quality  than  military  feculty  members,  with  over  half  of  the  respondents 
rating  civilian  feculty  as  either  “excellent”  or  “outstanding.”  However,  some  students 
were  not  even  exposed  to  civilian  instructors  and  therefore,  the  data  may  be  skewed.  Yet, 
written  comments  on  open-ended  questions  were  very  critical  of  military  feculty  members 
overall.  One  of  the  most  frequent  comments  on  the  open-ended  portion  of  the 
questionnaire  was  that  faculty  members  were  not  qualified  to  teach  the  subject  matter. 

Table  16: 

Faculty 

(in  percentages) 


Item 

Unsat 

(1) 

Marg 

(2) 

Sat 

(3) 

Excel 

(4) 

Out 

(5) 

Mean 

SD 

Military  Faculty 

3.1 

12.8 

39.8 

39.4 

4.9 

3.30 

.87 

Civilian  Faculty 

2.3 

8.1 

34.8 

44.3 

10.4 

3.53 

.87 

Faculty  Overall  Mean  =  3.37 
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Table  17  deals  with  students’  perceptions  about  the  number  of  Ph.D.s.  The  data  from 
this  table  show  that  respondents  were  moderately  in  fevor  of  adding  more  Ph.D.s  to  the 
feculty  with  over  half  of  respondents  (57,4  %)  replying  in  the  “very  little”  or  “to  some 
extent”  categories.  Again,  the  standard  deviation  of  1.05  indicates  a  wide  range  of  scores 
on  this  it^m.  Apparently,  students  were  -nDfin  total  agreement  about  the  extent  to  which 
Ph.D.s  should  be  added  to  the  faculty. 

Table  17; 

I  To  what  extent  should  the  ACSC  faculty  consist  of  more  Ph.D.  members?  I 


Not  at 

Very 

To  Some 

To  Considerable 

To  Great 

all  (1) 

little  (2) 

Extent  (3) 

Extent  (4) 

Extent  (5) 

2.7  % 

19.3  % 

38.1  % 

23.3  % 

16.6  % 

Mean  =  3.31  SD=1.05 


Table  18  and  19  summarize  data  about  faculty  balance  and  student/faculty  interaction. 
While  the  greatest  percentage  of  students  perceived  that  the  balance  between  civilian  and 
military  members  was  about  right  (64.9  %),  this  is  surprising  in  light  of  data  from  table  16 
in  which  respondents  rated  civilian  faculty  slightly  above  military  faculty.  The  issue  of 
studenfrfacuky  interaction  revolves  around  the  time  the  two  groups  spent  socializing  and 
getting  to  know  one  another.  According  to  data  in  table  19,  students  perceived  that 
enough  time  was  allotted  for  interacting  with  their  faculty  members  with  67.1  % 
responding  that  the  time  allotted  to  interaction  was  “about  right.” 


Table  18:  Faculty  Balance 


1  What  did  you  think  About  the  balance  between  civilian  I 

and  militaty  faculty  members? 

More  Military 

More  Civilian 

About  Right 

No  Response 

6.6  % 

25.  0  % 

64.9  % 

3.5  % 

Table  19: 

Student/Facultv  Interaction 
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1  What  did  you  think  about  the  time  allotted  for  I 

student/faculty  interaction? 

Too  Much 

Not  Enough 

About  Right 

No  Response 

.4% 

29.4  % 

67.1  % 

3.1  % 

Grading.  The  grading  category  revolved  around  grading  procedures,  feedback,  the 
process  by  which  students  challenge  their  grades  (reclama),  and  the  role  of  grades  in 
determining  distinguished  graduates  (items  66,  71.  72,  73,  and  80).  Table  20  displays  data 
on  grading  procedures.  As  is  evidenced  by  the  data,  nearly  fifty-two  percent  (5 1 .6  %)  of 
respondents  rated  grading  procedures  either  “unsatisfactory”  or  “marginal.”  This  was  the 
lowest  rated  item  described  thus  fer.  Obviously,  grading  procedures  at  ACSC  were 
perceived  as  less  than  adequate.  Student  open-ended  comments  claimed  similar 
sentiments  and  corroborate  the  low  rating  reflected  in  table  20.  Students  perceived  that 
grades  were  assigned  arbitrarily  and  were  not  standardized  throughout  the  College.  Of 
particular  concern  to  students  was  the  grading  procedures  for  their  research  projects.  A 
common  thread  in  students’  written  comments  was  how  the  lack  of  standardization  in 
grading  may  have  kept  many  people  from  becoming  distinguished  graduates. 

Table  20: 


Grading 

(in  percentages) 

Item 

Unsat 

(1) 

Marg 

(2) 

Sat 

(3) 

Excel 

(4) 

Out 

(5) 

Mean 

SD 

Grading  Procedures 

21.6 

30.0 

34.8 

13.2 

.4 

2,41 

.98 

Grading  Procedures  Overall  Mean  =  2.41 
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The  following  three  tables  (21-23)  cover  questions  71,  72,  and  73  on  the  questionnaire 
and  answer  more  specific  concerns.  Question  71  involved  feedback  on  oral  and  written 
work,  while  72  dealt  with  the  reclama  process.  Reclama  is  the  process  by  which  students 
can  protest,  through  official  channels,  a  particular  grade  in  a  course.  Finally,  question  73 
asked  students  about  the  extent  to  which  grades  should  be  used  in  determining 
distinguished  graduates. 

The  data  in  table  21  show  that  over  three-quarters  (78.4  %)  of  students  replied  that 
they  received  feedback  on  their  work  at  least  to  some  extent.  A  small  number  of  students 
(3.1  %)  claimed  that  they  did  not  receive  any  feedback. 

Table  22  reflects  students’  responses  on  the  reclama  process.  This  item  was  included 
because  senior  leaders  at  ACSC  were  interested  in  knowing  whether  or  not  students  were 
aware  of  options  available  to  them  if  they  should  want  to  dispute  a  grade.  Since  the 
graduates  fi*om  AY  1994-1995  were  only  the  second  class  to  fall  under  a  letter  grade 
system  in  current  ACSC  history,  leaders  at  the  college  felt  it  was  necessary  to  institute  a 
system  to  aid  students  in  questioning  a  grade  assigned  to  their  work.  Almost  seventy 
percent  (69.6  %)  revealed  that  they  knew  about  and  understood  the  process  at  least  to 
some  extent.  Clearly,  there  were  outliers,  with  a  small  number  of  students  responding 
“not  at  aU”  (5.4  %)  on  one  end,  and  a  small  amount  (8.5  %)  on  the  other  end  in  the  “to  a 
great  extent”  column.  This  was  also  reflected  in  the  standard  deviation  of  1 .04  which 
indicated  less  clustered  scores  and  a  lack  of  consensus  about  the  reclama  process.  It  is 
likely  that  those  who  used  the  process  knew  the  most  about  it,  while  those  who  did  not 
need  to  refute  grades,  knew  very  little. 
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Table  21: 

_^^c^age5f^ _ 

To  what  extent  did  you  receive  feedback  on  your  oral  and  written  work? 


Not  at 

Very 

To  Some 

To  Considerable 

To  Great 

all  (1) 

little  (2) 

Extent  (3) 

Extent  (4) 

Extent  (5) 

3.1  % 

18.6  % 

36.3  % 

35.0  % 

7.1  % 

Mean  =  3.24  SD  =  .94 


Table  22: 

To  what  extent  did  you  know  about  and  understand  the  reclama  process? 


Not  at 

Very 

To  Some 

To  Considerable 

To  Great 

all  (1) 

little  (2) 

Extent  (3) 

Extent  (4) 

Extent  (5) 

5.4  % 

25.0  % 

33.9  % 

27.2  % 

8.5  % 

Mean  =  3.09  SD=1.04 


Figures  in  table  23  summarize  the  data  about  the  use  of  grades  in  determining 
distinguished  graduates.  Over  one-quarter  (27.3  %)  of  graduates  felt  that  grades  should 
not  be  used  at  all  or  very  little  to  determine  distinguished  graduates.  On  the  other  end  of 
the  spectrum,  nearly  fourteen  percent  (13.9  %)  fek  they  should  be  used  “to  a  great 
extent.”  This  dichotomy  was  reflected  in  the  standard  deviation  of  1 .3  showing  large 
variability  in  the  range  of  scores  on  this  item.  This  seems  to  suggest  that  those  who 
became  distinguished  graduates  probably  believed  that  grades  should  be  used,  while  those 
who  did  not  make  it,  were  not  as  enthusiastic  about  the  system.  However,  some  students 
who  self-disclosed  on  the  open-ended  portion  of  the  questionnaire  that  they  were 
distinguished  graduates  were  critical  of  the  program  and  how  grades  were  used  as  a  fector 
of  determining  who  would  or  would  not  be  chosen. 
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Table  23: 

To  what  extent  should  grades  be  used  for  determining  distinguished  graduates? 


Not  at 

Very 

To  Some 

To  Considerable 

To  Great 

all(l) 

little  (2) 

Extent  (3) 

Extent  (4) 

Extent  (5) 

19.7  % 

7.6  % 

33.2  % 

25.6  % 

13.9  % 

Mean  =  3.06  SD=1.30 


The  last  item  in  the  grading  category  revolves  around  the  grading  method.  For  the  past 
two  years,  ACSC  has  been  issuing  grades  for4heir-co«rses  rather  than  recording  them  as 
pass/fail.  This,  of  course,  created  much  controversy  and  questions  about  the  benefit  of 
each  method.  When  students  were  asked  what  grading  method  they  thought  was  more 
appropriate  at  ACSC,  they  overwhelmingly  voted  for  pass/fail  grades.  The  breakdown  of 
this  is  recorded  in  table  24. 


Table  24: 
Grading  Method 


1  What  grading  method  do  you  think  is  more  I 

appropriate  at  ACSC? 

Letter  Grades 

Pass/Fail 

30.3  % 

66.7  % 

Clearly,  students  would  rather  receive  pass/tail  grades  than  letter  grades  for  their  efforts 
at  ACSC.  Yet,  it  is  difficult  to  know  what  they  might  have  said  if  they  had  experienced  a 
pass/feU  system.  Since  these  graduates  did  not  have  an  option,  they  could  not  be  objective 
about  the  value  of  pass/M.  However,  many  respondents  wrote  in  the  open-ended  portion 
of  the  questionnaire  that  by  having  letter  grades,  students  were  too  competitive  and 
focused  less  on  learning  than  on  getting  the  top  score. 
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Student  Perceptions  of  Competency  on  Outcome  Measures 

The  outcome  variables  consisted  of  the  following  four  categories: 

•  Joint  Campaign  Planning  Process 

•  Air/Space  Power  Applications 

•  Command  and  Leadership 

•  Critical  Thinking 

Descriptive  data  on  these  categories  were  compiled  from  student  responses  to  items  5-49 
on  the  questionnaire.  Students  were  asked:  “How  would  you  rate  your  conqjetency  on 
the  following  items  at  the  point  in  time  when  you  completed  Air  Command  and  Staff 
College?”  A  five-point  rating  scale  was  used  to  rate  their  perceptions  about  their 
competency  on  a  number  of  items.  These  items  were  compiled  through  a  content  analysis 
of  course  syllabi.  Course  objectives  were  extracted  from  each  syllabus  and  then  grouped 
into  “like”  categories.  The  four  areas  listed  above  emerged  as  a  result  of  this  analysis. 
Each  category  is  summarized  separately  by  grouping  appropriate  questions  together  to 
form  the  four  categories  mentioned  above. 

Joint  Campaign  Planning  Process.  The  joint  campaign  planning  process  is  one  of  the 
major  overarching  concepts  taught  at  ACSC.  It  involves  teaching  students  all  that  is 
involved  in  conducting  joint  (inter-service)  campaigns  much  like  the  Desert  Storm 
campaign.  Table  25  summarizes  the  results  of  the  descriptive  data  for  this  category  (items 
35-45  on  the  questionnaire). 
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Table  25: 


Competency  on  Joint  Campaign  Planning  Process 

(in  percentages) 

Item 

m 

Very 

little 

(2) 

Some 

(3) 

Consid 

-  (4)_ 

Mean 

SD 

Understanding  critical  factors  in 
planning  a  theater  military  operation. 

1.3 

3.1 

36.8 

48.2 

10.5 

3.64 

.77 

Understanding  structure  of  a  joint 
organization. 

.4 

6.6 

28.2 

48.9 

15.9 

3.73 

.82 

Understanding  Air  Force  role  in  joint 
planning. 

.9 

5.7 

27.2 

47.4 

18.9 

3.78 

.85 

Understanding  and  applying  joint 
operations  planning. 

1.8 

6.6 

39.0 

42.1 

10.5 

3.53 

.84 

Understanding  roles,  functions, 
capabilities,  &  limitations  of  US 
military  forces  in  joint  operations. 

1.3 

7.0 

32.9 

51.8 

7.0 

3.56 

.78 

.4 

2.6 

25.0 

51.8 

20.2 

3.89 

.77 

0 

3.1 

25.9 

53.5 

17.5 

3.86 

.73 

Understanding  basic  concepts  &  issues 
in  planning  &  executing  war 
termination. 

1.3 

4.4 

23.7 

51.3 

19.3 

3.83 

.84 

Understanding  political,  economic,  & 
military  roles  in  post-hostilities 
environment. 

1.3 

3.1 

23.2 

49.6 

22.8 

3.90 

.83 

Understanding  role  of  technology  to 
conduct  and  win  campaigns. 

.9 

.4 

18.4 

49.6 

30.7 

4.09 

.76 

Understanding  need  for  resource 
management  &  acquisition  to  support 
fijture  force  development. 

1.8 

3.9 

23.2 

50.0 

21.1 

3.85 

.86 

Joint  Campaign  Planning  Process  Overall  Mean  =  3.78  SD  =  .80 

The  data  on  the  table  show  that  students  rated  their  competency  on  most  items,  on  the 
average,  very  close  to  “considerable.”  The  lowest  rated  item  on  this  table  is 
“understanding  and  applying  joint  operations  planning”  with  a  mean  of  3.53,  but  even  on 
that  item,  over  fifty  percent  (52.6  %)  of  the  graduates  rated  their  competency  as  either 
“considerable”  or  “great.”  The  highest  rated  item  at  4.09  deals  with  the  role  of  technology 
in  conducting  and  winning  campaigns.  This  is  not  surprising  given  the  emphasis  on 
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technology  in  modem  warfare  and  the  technological  expertise  Air  Force  officers  are 
expected  to  possess.  Standard  deviations  for  all  items  are  also  very  constant  indicating  a 
relative  consensus  among  respondents  on  the  items. 

Air/Space  Power  Applications.  Table  26  displays  descriptive  statistics  for  this  second 
major  outcome  variable.  The  data  correspond  to  items  46-49  on  the  questionnaire. 

Table  26: 


Stadm^lPCTcegt^s^fC^ggOT^_OT_Air/SgaceP^raAgglkat^s_ 


Competency  on  Air/Space  Power  Applications 

(in  percentages) 

Item 

None 

(1) 

Very 

little 

(2) 

Some 

(3) 

Consid 

(4) 

Great 

(5) 

Mean 

SD 

Understanding  role  &  impact  of 
technology  on  air  campaign  planning. 

A 

2.2 

14.5 

56.6 

26.3 

4.06 

.73 

Understanding  contributions  of  air  and 
space  power  on  air  campaign  planning. 

0 

.9 

29.8 

4.13 

.68 

Understanding  &  applying  air 
campaigning  to  national  scenarios. 

.9 

1.8 

26.3 

50.9 

20.2 

3.88 

.78 

Understanding  &  analyzing  impact  of 
contextual  &  operational  art  elements 
on  military  campaigns. 

.9 

2.6 

15.4 

49.1 

32.0 

4.09 

.81 

Air/Space  Power  Applications  Overall  Mean  =  4.04  SD  =  ,75 

Clearly,  students  rated  their  competency  in  this  category  relatively  high  with  over 

three-quarters  of  respondents  claiming  they  have  “considerable”  or  “great”  competence  in 
three  of  the  four  items.  The  lowest  rated  item  was  “understanding  and  applying  air 
campaigning  to  national  scenarios”  but  even  on  that  item,  seventy-one  percent  (71  %)  of 
graduates  responded  that  they  possessed  “considerable”  or  “great”  competence.  More 
respondents  (26.3  %)  replied  in  the  “some  competence”  column  for  that  item  than  the 
others  but  this  does  not  indicate  a  lack  of  perceived  competence.  It  is  interesting, 
however,  that  on  two  of  the  highest  rated  items  (#1  and  #4),  a  small  number  of  graduates 
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responded  in  the  “very  little”  competence  column.  These  figures  stand  out  because  of  the 
high  overall  rating  students  gave  on  those  items.  One  must  wonder,  however,  if  the  low 
rating  stems  fi'om  the  vague  wording  of  the  item  rather  than  a  rating  based  on 
competency. 

Command  and  Leadership.  This  category  consisted  of  items  5-16  on  the  questionnaire. 
Table  27  summarizes  the  results  of  the  data  analysis. 

Table  27: 


S^ents^^^^tiOTS_of£^gg|^c^^_C^m^d_^dLea4CT^i^^i^m 


Competency  on  Command  and  Leadership 

(in  percentages) 

Item 

H^sjll 

Some 

(3)  . 

Consid 

. (4) . 

Great 

(5) 

Mean 

SD 

Comparing  personality  types  to  make 
decisions. 

1.8 

12.8 

41.9 

38.3 

5.3 

3.33 

.83 

Understanding  &  applying  leadership 
skills  in  diverse  situations. 

.4 

4.8 

28.9 

54.8 

11.0 

3.71 

.74 

Understanding  &  applying  counseling 
techniques  in  a  variety  of  situations. 

2.6 

14.0 

43.9 

31.6 

7,9 

3.28 

.90 

Applying  principles  of  oral  &  written 
communication  in  different  situations. 

0 

.9 

53.9 

27.6 

4.08 

.69 

Understanding  Air  Force  officer 
promotion  system. 

2.2 

3.9 

26.3 

42.5 

25.0 

3.84 

.92 

Understanding  &  applying  quality 
force  tools  and  techniques. 

3.1 

16.2 

48.2 

28.9 

3.5 

3.14 

.84 

Understanding  the  special  nature  of 
the  military  leader. 

.9 

■■ 

21.9 

51.3 

21.5 

3.88 

.82 

1.3 

11.8 

54.8 

jjgm 

4.08 

Understanding  combat  leader  styles 
throughout  histoiy^ 

.4 

23.7 

52.6 

3.86 

■1 

Understanding  inclusion  of  strong 
moral  character  &  ethical 
considerations  in  decision-making. 

.4 

2.6 

12.3 

45.2 

39.5 

4.21 

.79 

Understanding  commanders’  role  & 
responsibilities 

.4 

3.9 

25.0 

^gnnn 

21.5 

3.87 

.81 

Understanding  key  elements  of  Army, 
Navy,  Air  Force,  &  Marine  officer 
evaluation  system. 

21.5 

52.3 

2.2 

2.97 

Command  and  Leadership  Overall  Mean  =  3.69  SD  =  .81 
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As  can  be  seen  by  looking  at  the  table,  most  items  fell  very  close  to  “considerable”  on 
the  five-point  scale.  The  lowest  rated  item  involves  officer  evaluation  systems  for  all  the 
services  with  over  three-quarters  (77.3  %)  rating  their  competence  as  “some,”  “very 
little,”  or  “none.”  The  highest  rated  item  concerns  including  moral  character  and  ethics  in 
decision-making.  Nearly  eighty-five  percent  (84.7  %)  rated  their  competence  on  that  item 
as  “considerable”  or  “great.”  Item  4  on  the  table,  “applying  principles  of  oral  &  written 
communication  in  different  situations”  is  noteworthy.  Although  it  was  not  the  highest 
rated  item,  students  rated  their  competence,  on  the  average,  as  “considerable”  with  a  low 
standard  deviation  of  .69,  indicating  that  there  was  little  disagreement  about  their 
perceived  competency  on  that  item.  While  respondents,  on  the  average,  perceived  their 
competency  on  Command  and  Leadership  as  being  close  to  “considerable,”  it  is  interesting 
that  some  students  did  not  feel  confident  in  their  competence  in  this  area  as  evidenced  by 
the  number  of  respondents  who  responded  in  the  “very  little”  and  “none”  columns.  If  one 
compares  these  figures  to  those  in  the  other  outcome  variables,  one  can  see  a  dramatic 
difference.  Some  students  criticized  ACSC’s  handling  of  the  Command  and  Leadership 
category  in  the  open-ended  section  of  the  questionnaire.  Typical  comments  stated  that  not 
enough  time  in  the  curriculum  was  devoted  to  Command  and  Leadership  while  others 
criticized  the  topics  that  were  emphasized. 

Critical  Thinkmg.  The  analysis  for  the  Critical  Thinking  category  was  based  on  items 
17-34  on  the  questionnaire.  Table  28  shows  results  of  the  data  analysis. 
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Table  28: 


Students’  Perceptions  of  Competency  on  Critical  Thinking  by  item 


Competency  on  Critical  Thinking 

(in  percentages) 

Item 

m 

Very 

little 

(2) 

Some 

(3) 

Consid 

(4) 

Mean 

SD 

Challenging  assumptions 

1.3 

2.6 

49.8 

24.7 

3.94 

.83 

Breaking  down  barriers  to  creative 
thinking. 

.9 

3.1 

32.5 

46.5 

17,1 

3.76 

.80 

Translating  ideas  into  action. 

1.3 

.9 

21.5 

57.9 

18.4 

3.91 

Understanding  &  implementing 
change  in  a  military  environment. 

1.3 

4.8 

31.1 

50.0 

12.7 

3.68 

■I 

Understanding  differences  between 
conflict  and  war. 

.4 

3.9 

22.8 

52.6 

20.2 

ngm 

.79 

Understanding  social  aspect  of  war. 

.9 

1.8 

20.2 

59.2 

18.0 

.73 

Understanding  circumstances  of  war. 

.9 

18.5 

59.9 

jESM 

.69 

Understanding  &  applying  war  & 
conflict  concepts  in  different 
circumstances. 

.4 

.9 

22.0 

56.8 

.70 

Understanding  &  creating  war  theory. 

.4 

5.3 

28.2 

49.8 

16.3 

■SBSI 

hihi 

Understanding  relationship  of  war 
theory  to  practice  of  waging  war. 

.9 

21.9 

55.3 

19.3 

Understanding  interrelationship  of 
historical  experience  and  development 
of  war  theory. 

.9 

18.1 

55.9 

23.3 

4.00 

.75 

0 

2.6 

23.7 

50.0 

23.7 

4.00 

.76 

Understanding  how  to  influence 
international  actors  to  achieve  national 
objectives. 

0 

2.2 

29.8 

53.9 

14.0 

3.80 

.70 

Identifying  &  analyzing  a  military 
force’s  center  of  gravity. 

0 

11.4 

44.7 

42.1 

4,27 

.73 

Understanding  military  instrument  of 
national  power. 

.4 

.9 

7.0 

49.6 

42.1 

4.32 

.68 

Synthesizing  new  ideas  on  application 
of  air  and  space  power. 

3.5 

34.2 

46.9 

14.9 

3.72 

.77 

Analyzing  &  applying  war  termination 
concepts  in  current  &  potential 
scenarios. 

1.3 

3.5 

28.9 

54.4 

11.8 

3.72 

.77 

Understanding  role  of  innovation  in 
superior  war  fighting  capability. 

1.3 

53.9 

21.5 

3.93 

.79 

Critical  Thinking  Overall  Mean  =  3.90  SD  =  .76 

From  this  table  one  can  see  that  students,  on  the  average,  perceived  their  competencies 

on  the  Critical  Thinking  variable  as  “considerable.”  The  lowest  rated  item  was 
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“understanding  &  implementing  change  in  a  military  environment”  but  even  so,  almost 
sbcty-three  percent  (62.7%)  still  rated  their  competencies  as  “considerable”  and  “great.” 
The  standard  deviation  on  this  item,  although  higher  than  most  in  the  category,  does  not 
show  a  great  deal  of  variability  indicating  that  the  scores  are  clustered  close  together. 

The  highest  rated  item  was  “understanding  the  military  instrument  of  national  power” 
which  reveals  that  almost  ninety-two  percent  (91 .7%)  of  the  graduates  rated  their 
competencies  as  either  “considerable”  or  “great.”  The  data  in  table  28  shows  that 
students  rated  themselves  relatively  high  on  all  items  and  that  there  is  little  disagreement 
among  them  about  the  ratings,  based  on  standard  deviations. 

Relational  Analysis 

The  specific  relational  questions  posed  in  chapters  one  and  three  of  this  dissertation  were: 
“Was  there  a  relationship  between  input  variables  (demographics  and  student 
expectations)  and  outcome  variables?”  and  “Was  there  a  relationship  between 
environmental  variables  (teaching  methods  and  program  activities)  and  student 
outcomes?”  To  answer  the  first  question,  the  cross  tabulation  method  was  utilized  to 
determine  the  relationship  between  demographics/  student  expectations  and  outcomes. 
The  data  were  reported  in  percentages  and  represented  the  number  of  respondents  who 
perceived  a  high  competence  (an  average  of  4  and  5)  on  outcome  variables  by  each 
demographic  category  (age,  race,  gender,  etc.).  Question  number  two  was  answered 
through  correlational  methods  and  results  were  reported  as  Pearson  correlational 


coefficients  (Pearson  “r”). 
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The  Relationship  Between  Input  Variables  and  Outcome  Variables 

Demographic  characteristics  of  graduates  consisted  of  six  categories  (age,  gender, 
race,  career  field,  Master  degree  concentration,  and  commission  source).  Items  82-87  on 
the  questionnaire  show  each  category  and  their  sub-categories.  Student  expectations 
(question  #88)  consisted  of  nine  statements  in  which  students  were  asked  to  check  all  that 
apply  to  them.  There  are  four  outcome  variables  (Joint  Campaign,  Air/Space  Power, 
Command  and  Leadership,  and  Critical  Thinking).  Items  5-49  on  the  questionnaire 
reflected  outcome  items  divided  among  the  four  variables  as  follows: 

•  Joint  Campaign  Items  35-45 

•  Air/Space  Power  Items  46-49 

•  Command  and  Leadership  Items  5-16 

•  Critical  Thinking  Items  17-34 

For  the  purpose  of  this  analysis,  the  individual  items  that  made  up  each  outcome  variable 
were  added  together  to  create  a  single  value  for  each  outcome  variable.  For  example, 
items  35-45  were  added  together  to  create  the  Joint  Campaign  variable.  These  values 
were  referred  to  as  “collapsed”  values.  Collapsed  values  were  based  on  respondents’  total 
scores,  with  the  average  score  being  “4”  and  “5.”  These  values  reflected  those  who 
perceived  their  competence  as  being  high  based  on  a  total  score  obtained  by  adding 
together  individual  items.  While  the  relational  analysis  was  based  on  collapsed  values  and 
those  values  reported  in  this  chapter,  data  on  the  relationships  between  individual  items 
and  outcome  variables  were  reported  in  the  appendbc  (Appendbc  C).  The  association 
between  individual  items  and  outcome  variables  also  reflected  those  who  perceived  their 
competence  as  being  high  but  high  values,  in  this  case,  were  based  on  the  number  of 
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respondents  who  rated  their  competencies  “considerable”  (4)  and  “great”  (5)  rather  than  a 
total  score.  Please  note  that  when  comparing  collapsed  values  on  outcome  variables  with 
each  of  the  items  that  made  up  the  variables,  collapsed  values  were  smaller.  For  example, 
the  collapsed  percentages  on  Joint  Campaign  presented  in  the  tables  contained  in  this 
chapter  were  smaller  when  compared  to  the  average  score  for  each  of  the  items  that  made 
up  the  scale  (see  Appendix  C).  Because  collapsed  values  were  based  on  a  total  score 
rather  than  on  just  the  number  of  individuals  scoring  “4”  and  “5”,  the  number  of 
individuals  scoring  high  on  collapsed  variables  was  decreased,  resulting  in  lower 
percentages. 

Age  and  Outcome  Variables.  Table  29  shows  the  percentage  of  respondents  in  each 
age  category  who  rated  their  competency  on  the  average  as  “considerable”  or  “great.” 

Tabic  29: 

Percentage  of  Respondents  Scoring  High  (an  average  of4  and  5)  on  Outcome 

Variables  bv  Age. _ 


Rate  (%)  Amou] 

a 

Outcome 

Variable 

35  &  under 
n  =  54 

36-39 
n  =  115 

40  &  over 
n  =  51 

Joint  Campaign 

33%  (18) 

48  %  (55) 

31  %(16) 

Air/Space  Power 

59  %  (32) 

73  %  (84) 

57  %  (29) 

Command  &  Leadership 

28%  (15) 

32  %  (37) 

33  %(17) 

Critical  Thinking 

37  %  (20) 

56  %  (64) 

33  %  (17) 

According  to  the  data  in  the  table,  the  Air/Space  Power  outcome  variable  was  the  only 
one  where  a  solid  majority  of  the  respondents  reported  having  high  competence.  The 
lowest  percentage  of  respondents  scoring  high  occurs  in  the  Command  and  Leadership 
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variable.  Overall,  the  largest  percentages  of  respondents  scoring  high  on  outcome 
variables  was  in  Age  2  (36-39  age  group),  except  for  Command  and  Leadership,  which,  at 
a  modest  thirty-two  percent  (32  %),  is  not  much  different  than  the  other  age  categories. 

A  more  in-depth  look  at  student  perceptions  of  their  competencies  on  individual  items 
(please  see  Appendk  C-1)  by  age  reveals  more  detailed  information.  The  lowest 
percentages  of  respondents  scoring  high  was  on  item  16,  a  Command  and  Leadership 
item,  “understanding  key  elements  of  the  Army,  Navy,  Air  Force,  and  Marine  officer 
evaluation  systems.”  The  highest  percentages  on  which  respondents  perceived  high 
competence  was  item  3 1 ,  “Understanding  the  military  instrument  of  national  power,”  a 
Critical  Thinking  item.  When  comparing  the  items  by  age,  it  is  apparent  that  the  largest 
percentages  occurred  in  the  36-39  age  group  on  most  individual  items.  In  Joint 
Campaign,  the  difference  between  student  perceptions  of  competencies  in  Age  2  (36-39 
age  group)  and  the  other  two  groups  is  close  to  thirty  percentage  points  in  some  cases. 

The  difference  narrows  in  Air/Space  Power  and  Command  and  Leadership,  but  widens 
again  in  Critical  Thinking.  For  the  most  part,  the  percentages  of  high  competence  for  age 
category  36-39  were  higher  than  the  other  age  groups  on  all  items.  On  two  items  (9  and 
1 1),  however,  the  percentages  of  high  competence  for  ages  36-39  were  considerably  lower 
than  the  other  age  groups.  These  items  deal  wdth  understanding  the  officer  promotion 
system  (9),  and  the  special  nature  of  the  military  leader  (11).  Perhaps  those  in  the  36-39 
age  group  perceived  their  competencies  less  in  these  particular  items  because  they 
expected  more  information  about  how  the  promotion  system  works  and  about  how  to  be  a 
better  leader.  It  is  likely  that  older  officers  feel  less  anxious  about  promotions  and  their 
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leadership  potential  because  they  have  had  more  life  experiences,  while  younger  officers 
may  have  more  feith  in  their  abilities  to  compete. 


Gender  and  Outcome  Variables.  Table  30  reflects  the  percentages  of  respondents 
scoring  high  on  outcome  variables  for  both  males  and  females. 

Table  30: 

Percentage  of  Respondents  Scoring  High  (an  average  of  4  and  5) 
on  Outcome  Variables  bv  Gender. 


Rate  (%)  Among 

Outcome  Variable 

Male 
n  =  179 

Female 
n  =  41 

Joint  Campaign 

38  %  (68) 

51  %(21) 

Air/Space  Power 

66%  (118) 

98  %  (40) 

Command  &  Leadership 

31  %(55) 

34  %  (14) 

Critical  Thinking 

47  %  (84) 

41  %(17) 

The  table  shows  that  female  respondents  scored  their  competency  as  high,  as  compared 
to  male  respondents,  on  three  of  the  four  outcome  variables.  Apparently,  female 
respondents  were  more  confident  in  their  abilities  than  males  were  in  all  variables  except 
Critical  Thinking.  The  largest  percentages  of  respondents  reported  high  competence  in 
the  Air/Space  Power  outcome  variable  and  the  smallest  in  the  Command  and  Leadership 
variable. 

A  review  of  individual  item  differences  (Appendix  C-2)  shows  similar  patterns  as  in  the 
age  category.  For  example,  the  overall  lowest  percentage  of  respondents  scoring 
themselves  high  on  outcomes  was  on  item  1 6,  and  the  overall  highest  was  item  31.  On  the 
average,  the  differences  between  male  and  female  percentages  were  not  large.  However, 
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on  three  items,  the  differences  stood  out  and  are  worthy  of  mention.  On  item  43, 
“understanding  the  political,  economic,  and  military  roles  in  post-hostilities  environment,” 
at  85%,  the  percentage  of  female  respondents  scoring  their  competency  as  high  was 
sixteen  (16)  percentage  points  higher  than  male  respondents  at  69%.  Conversely,  on  items 
1 1  and  25,  the  percentages  of  males  scoring  their  competencies  as  high  were  as  much  as 
twenty-two  (22)  percentage  points  higher  than  females.  Item  1 1  deals  with  the  special 
nature  of  the  military  leader,  while  item  25  concerns  understanding  and  creating  war 
theory.  One  possible  explanation  for  the  differences  revolves  aroimd  typical  male/female 
stereotypes.  While  females  may  feel  more  confident  in  their  abilities  to  understand  and 
interpret  theories,  males  may  feel  more  at  ease  with  creating  them,  especially  where 
military  leadership  and  war  theory  are  concerned.  Another  possible  reason  for  the 
difference  in  perceived  competence  levels  by  gender  may  relate  to  the  career  fields 
available  to  men  and  women.  Until  recently,  women  were  excluded  fi'om  combat-related 
jobs,  which  included  flying  fighter  aircraft.  Perhaps  women  feel  less  competent  in  items 
relating  to  military  leadership  and  war  theory  because,  in  the  past,  they  have  not  been  full 
participants  in  the  mission  of  the  Air  Force. 

Race  and  Outcome  Variables.  Table  31  summarizes  the  percentages  of  respondents 
scoring  their  competence  as  high  on  outcome  variables  by  race.  Although  some  race 
categories  are  small,  such  as  Asian,  Hispanic,  and  Native-American,  they  are  included 
because  even  though  they  represent  a  small  number  of  graduates,  their  perceptions  are  no 
less  important. 
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Table  31  : 

^COTtageof^gOTdOTts^COTm£K^lMj^ageof4_^d5)_OT^^c^£^m^les^^^^^ 


Rate  (%)  Among 

Outcome  Variable 

African- Am 
n  =  17 

Asian 
n  =  4 

Hispanic 
n  =  10 

Native- Am 
n  =  4 

White 
n  =  181 

Joint  Campaign 

65%  (11) 

25  %  (1) 

40  %  (4) 

25  %  (1) 

39  %  (70) 

Air/Space  Power 

82  %  (14) 

50  %  (2) 

70  %  (7) 

75  %  (3) 

64%  (116) 

Command  &  Leadership 

35  %  (6) 

25  %  (1) 

50  %  (5) 

50  %  (2) 

30  %  (54) 

Critical  Thinking 

47  %  (8) 

50  %  (2) 

50  %  (5) 

75  %  (3) 

45%  (81) 

The  table  reveals  that  out  of  the  five  race  categories,  the  largest  percentages  of 
respondents  who  perceived  their  competence  as  high  were  Afi'ican- Americans,  except  in 
Command  and  Leadership  and  Critical  Thinking.  The  Hispanic  and  Native- American  race 
categories  follow  African-Americans  on  overall  perceived  competency,  and  then  the  White 
race  category.  The  lowest  percentages  of  overall  perceived  competence  was  the  Asian 
race  category  except  in  Critical  Thinking,  where  Asian  respondents  scored  their 
competency  higher  than  either  Afi’ican- Americans  or  White  respondents.  The  largest 
percentages  of  respondents  scoring  high  by  race  categories  occurred  in  the  Air/Space 
Power  variable.  The  smallest  percentages  of  perceived  competency  varied  depending  on 
the  race  category.  For  example,  the  least  percentage  of  Afi’ican-American,  Asian,  and 
White  respondents  rated  their  competencies  high  in  the  Command  and  Leadership 
variable,  while  the  lowest  percentages  of  Hispanic  and  Native-American  respondents 
rating  themselves  high  was  in  the  Joint  Campaign  variable. 
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The  above  discussion  on  the  comparisons  that  can  be  made  from  looking  at  table  3 1 
follow  through  when  comparing  individual  items  by  race  category  (Appendix  C-3).  The 
largest  percentages  of  African-American  respondents  scored  themselves  high  on  most 
items.  However,  an  item-by-item  comparison  shows  that,  on  some  items,  Asian,  Hispanic, 
and  Native-American  graduates  scored  their  competencies  as  high.  It  is  interesting  to  look 
at  how  many  items  show  percentages  of  100%  for  those  race  categories,  but  the  reader 
should  be  cautioned  that  these  figures  are  based  on  small  sub-populations.  The  smaller  the 
sub-population,  the  greater  the  chance  of  having  aU  the  individuals  rating  themselves  as 
highly  competent.  As  in  previous  demographic  categories,  the  lowest  percentages  of 
respondents  scoring  high  by  race  were  in  item  16,  and  the  highest  were  in  item  31 , 
However,  the  percentages  of  African- Americans  scoring  themselves  as  highly  competent 
are  high  on  a  number  of  items.  Items  45,  46,  and  47  join  item  3 1  with  ninety-four  percent 
(94%)  of  African-American  respondents  rating  their  competencies  as  high.  Item  45,  a 
joint  campaign  item,  deals  with  resource  management  and  acquisition,  while  items  46  and 
47,  both  Air/Space  Power  items,  cover  the  role  and  impact  of  technology  on  air  campaign 
planning  and  understanding  the  contributions  of  air  and  space  power  to  theater  campaigns. 
The  lowest  percentage  (next  to  item  16)  of  African-Americans  scoring  their  competency 
as  high  was  on  item  7,  “understanding  and  applying  counseling  techniques  in  a  variety  of 
situations.”  While  the  reasons  for  these  high  and  low  percentages  are  not  immediately 
apparent,  it  is  probably  unlikely  that  there  are  racial  differences  that  created  the  high  and 
low  perceived  competencies  for  these  particular  items.  It  may  have  more  to  do  with 
career  fields  in  which  Afiican- American  officers  work.  Since  this  study  did  not  look  at  the 
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relationship  between  race  and  career  field,  it  is  difficult  to  speculate  about  this.  The 
lowest  percentage  of  White  respondents  rating  their  competency  as  high  occurred  in  item 
10,  “understanding  and  applying  quality  force  tools  and  techniques,”  and  the  most  on  item 
30,  “identifying  and  analyzing  a  military  force’s  center  of  gravity.”  Again,  it  is  difficult  to 
speculate  as  to  why  White  respondents  rated  their  competencies  high  or  low  on  certain 
items.  As  stated  earlier,  it  may  have  more  to  do  with  the  career  fields  in  which  White 
respondents  work  than  it  does  with  their  race. 

Career  Field  and  Outcome  Variables.  Table  32  shows  percentages  of  those  scoring 
high  on  outcome  variables  by  career  field.  The  career  fields  shown  are  Operations, 
Mission  Support,  Logistics,  and  Medical/Legal.  The  Operations  career  field  includes 
pilots,  navigators,  missileers,  air  traffic  controllers,  and  air  weapons  controllers.  Support 
officers  include  information  management,  personnel,  and  finance  officers.  Logistics 
officers  consist  of  those  in  acquisition  and  maintenance  career  areas. 

Table  32 : 


Percentages  of  Respondents  Scoring  High  (an  average  of  4  and  5)  on  Outcome  Variables 
by  Career  Field. _ 


Rate  (%)  Among 

Outcome  Variable 

Ops 
n  =  82 

Support 
n  =  75 

Logistics 
n  =  42 

Med/Legal 
n  =  7 

Joint  Campaign 

33  %  (27) 

39  %  (29) 

45  %  (19) 

71  %  (5) 

Air/Space  Power 

68  %  (56) 

63  %  (47) 

55  %  (23) 

100  %  (7) 

Command  &  Leadership 

27  %  (22) 

33  %  (25) 

33  %  (14) 

57  %  (4) 

Critical  Thinking 

44  %  (36) 

48  %  (36) 

74%  (31) 

86  %  (6) 
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Overall,  a  higher  percentage  of  officers  in  the  medical/legal  career  field  scored 
themselves  highly  competent  on  all  outcome  variables.  However,  the  percentages  were 
based  on  a  very  small  sub-population  (n=7)  which  increased  the  potential  for  all 
respondents  in  that  category  to  rate  themselves  as  highly  competent.  The  largest 
percentage  of  officers  who  perceived  their  competence  as  high  after  Medical/Legal  were 
Logistics  officers.  Logistics  officers  perceived  their  competence  in  the  Critical  Thinking 
outcome  variable  as  the  highest,  and  lowest  in  Command  and  Leadership.  Excluding  the 
Logistics  career  field,  more  respondents  scored  themselves  as  highly  competent  in  the 
Air/Space  Power  variable.  The  surprising  feet  about  the  data  in  table  32  are  the  low 
scores,  relative  to  the  other  career  fields,  that  appear  in  the  Operations  column.  On  the 
average,  the  percentages  of  officers  in  the  Operations  career  field  who  scored  their 
competence  as  high  were  the  lowest  among  the  four  areas  represented.  One  might 
speculate  that  this  is  because  of  the  nature  of  their  career  field.  Officers  who  have  spent 
many  years  in  the  “cockpit”  may  rate  their  competencies  low  in  areas  like  Command  and 
Leadership  because  they  do  not  directly  get  involved  in  commanding  others  (at  least  not  at 
their  rank  levels).  Similarly,  unless  pilots,  navigators,  and  other  operators  have  been  in 
joint  assignments,  they  may  not  perceive  themselves  as  being  highly  competent  in  Joint 
Campaign.  In  addition,  while  one  might  assume  that  pilots,  navigators,  and  other 
operators  would  perceive  themselves  as  highly  competent  in  critical  thinking  skills,  this  is 
not  necessarily  so.  Most  operators  rely  heavily  on  check  lists  in  almost  every  situation, 
including  emergencies.  This  might  explain  why  the  percentage  of  those  in  the  Operations 
career  field  scoring  high  on  Critical  Thinking  was  the  lowest  among  career  fields.  Not 
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surprisingly,  the  greatest  percentage  of  those  in  the  Operations  career  field  rated 
themselves  highly  competent  in  the  Air/Space  Power  variable.  Because  they  are  directly 
involved  in  air  and  space  power  issues  on  a  daily  basis,  they  may  have  more  confidence  in 
their  competencies  in  this  variable. 

An  in-depth  look  at  individual  outcome  variable  items  by  career  field  corroborates  the 
information  summarized  in  table  32  (see  Appendix  C-4).  Percentages  in  the 
Medical/Legal  column  are  the  highest  with  many  ratings  of  100%.  The  lowest  number 
(afl:er  item  16)  of  Medical/Legal  officers  scoring  themselves  highly  competent  on  outcome 
variables  was  on  item  5,  “comparing  different  personality  types  and  using  them  to  make 
decisions.”  While  the  overall  largest  percentage  of  Logistics  officers  who  rated 
themselves  highly  competent  occurs  in  the  Critical  Thinking  variable,  the  largest  niunber 
of  Logistics  officers  who  perceived  themselves  as  being  highly  competent  was  on  item  14, 
“understanding  the  inclusion  of  a  strong  moral  character  and  ethical  considerations  in 
decision-making,”  a  Command  and  Leadership  item.  The  percentages  on  individual  items 
compared  across  career  fields  are  very  close  between  Operations  and  Support  with  the 
largest  difference  being  no  more  than  ten  (10)  percentage  points.  The  lowest  percentage 
of  Operators  scoring  high  on  outcome  variables  (after  item  16),  was  on  item  5  on  using 
personality  types.  Although  percentages  fi’om  table  32  show  that  the  least  number  of 
officers  scoring  themselves  highly  competent  in  outcome  variables  was  in  the  Operations 
career  field,  a  comparison  of  individual  items  reveals  several  items  on  which  Operators 
scored  themselves  highly  competent  relative  to  other  career  fields.  The  largest  number 
(next  to  item  3 1 )  of  those  in  the  Operations  career  field  who  rated  themselves  highly 
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competent  was  on  item  27,  “understanding  the  interrelationship  of  historical  experience 
and  the  development  of  war  theory,”  a  Critical  Thinking  item.  The  strong  association 
between  Operations  and  item  27  may  relate  to  the  fact  that  pilots,  navigators  and  other 
operators’  roles  revolve  around  directly  conducting  war.  Another  possible  explanation  for 
this  association  may  relate  to  gender.  Operations  officers  tend  to  be  male,  and  male 
officers  perceived  themselves  as  highly  competent  on  items  concerning  war  theory  (see 
table  C-2  and  the  discussion  on  gender  and  outcome  variables). 

Master  Degree  Concentration/Education  Level  and  Outcome  Variables.  Tables  33 
and  34  show  the  percentages  of  respondents  scoring  high  on  outcome  variables  by  Master 
Degree  concentration  and  Education  Level.  Table  33  reveals  that  the  percentages  of 
respondents  scoring  high  on  competence  were  higher  for  those  with  Medical/Legal 
degrees  than  those  with  other  concentrations. 


Table  33 : 

Percentage  of  Respondents  Scoring  High  (an  average  of  4  and  5)  on  Outcome  Variables  bv  Master 
Degree  Concentration _ 


Rate  (%)  Among 

Outcome  Variable 

Humanities 
n  =  ll 

Soc  Science 
n  =  108 

Math/Sci 
n  =  52 

Engineering 
n  =  34 

Med/Legal 
n  =  6 

Joint  Campaign 

55  %  (6) 

41  %(44) 

40%  (21) 

26  %  (9) 

67  %  (4) 

Air/Space  Power 

73  %  (8) 

60  %  (65) 

81  %  (42) 

62%  (21) 

100  %  (6) 

Command  &  Leadership 

18%  (2) 

35  %  (38) 

29%  (15) 

21  %(7) 

67%  (4) 

Critical  Thinking 

45  %  (5) 

46  %  (50) 

46  %  (24) 

47  %  (16) 

83  %  (5) 
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Again,  the  number  of  individuals  in  the  Medical/Legal  category  was  small  increasing 
the  probability  that  all  of  those  with  MedicaVLegal  concentrations  will  score  high. 
Excluding  the  Medical/Legal  column,  the  largest  percentages  of  respondents,  on  the 
average,  with  perceived  high  competence  in  outcome  variables  were  in  the  Math/Science 
column.  The  largest  percentage  of  those  with  Math/Science  degrees  scoring  themselves 
highly  competent  was  on  the  Air/Space  Power  variable  at  eighty-one  percent  (81  %),  and 
the  lowest  percentage  on  the  Command  and  Leadership  variable  (29  %).  For  all  five 
Master  Degree  concentrations,  the  lowest  percentage  of  respondents  scoring  themselves 
highly  competent  was  in  the  Command  and  Leadership  variable  for  those  with  Humanities 
Degrees  (18  %).  Across  the  five  Master  Degree  concentrations,  the  percentages  of 
respondents  who  perceived  themselves  as  highly  competent,  fi*om  largest  to  smallest,  were 
in  Air/Space  Power,  Critical  Thinking,  Joint  Campaign,  and  Command  and  Leadership.  It 
is  not  surprising  that  the  largest  percentages  of  respondents  scoring  themselves  as  highly 
competent  occurred  in  the  Air/Space  Power  variable.  However,  it  is  interesting  that  the 
largest  percentage  of  respondents  scoring  high  on  competence  on  Air/Space  Power, 
excluding  Medical/Legal,  was  in  the  Math/Science  concentration.  Perhaps  this  was 
because  those  with  math  and  science  backgrounds  perceived  themselves  as  having  a 
greater  understanding  about  the  technical  aspects  of  air  and  space  power  principles.  Yet, 
if  this  were  true,  one  would  expect  those  with  engineering  degrees  to  have  perceived 
themselves  with  a  much  higher  competence  in  Air/Space  Power  as  well.  The  lowest 
percentages  across  degree  concentrations  was  in  the  Command  and  Leadership  variable, 
with  the  lowest  percentage  of  respondents  scoring  themselves  as  highly  competent  in  the 
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Humanities  column.  One  might  conclude  that  perhaps  this  was  due  to  gender.  However, 
out  of  the  eleven  (11)  respondents  with  Humanities  degrees,  only  four  (4)  were  females. 

A  more  likely  explanation  was  that  those  with  Humanities  degrees  were  not  in  jobs  in 
which  their  command  and  leadership  skills  were  utilized. 

Table  34  summarizes  percentages  of  respondents  scoring  high  on  outcome  variables  by 
Education  Level.  A  small  percentage  (n=4)  came  to  ACSC  without  an  earned  Master 
degree.  On  the  other  end  of  the  spectrum,  two-thirds  more  (n=13)  brought  with  them 
earned  Ph.D.  degrees. 


Table  34 : 

Percentages  of  Respondents  Scoring  High  (an  average  of  4  and  5)  on  Outcome  Variables 
bv  Education  Level. _ _ 


Rate  (%)  Among  I 

Outcome  Variable 

No  Master 
n  =  4 

Master 

n=211 

Master  + 
n  =  16 

Ph.D. 
n  =  13 

Joint  Campaign 

25%(1) 

40  %  (84) 

38  %  (6) 

31  %(4) 

Air/Space  Power 

75  %  (3) 

67  %  (142) 

69%  (11) 

69  %  (9) 

Command  &  Leadership 

25  %  (1) 

Critical  Thinking 

75  %  (3) 

47%  (100) 

75  %  (12) 

mm 

These  data  suggest  that  larger  percentages  of  those  with  no  Master  degree  perceived 
their  competency  as  high  on  the  four  outcome  variables  as  compared  to  those  with  Ph.D.s. 
Again,  since  the  number  of  individuals  in  the  sub-population  of  no  Master  degree  is 
relatively  small,  some  caution  must  be  taken  when  interpreting  these  percentages. 
However,  in  comparing  the  four  categories,  it  is  evident  that  the  largest  percentages  of 
respondents  scoring  themselves  highly  competent  in  outcomes  were  those  with  Master 


118 


Plus.  While  one  might  expect  the  largest  percentages  of  respondents  who  scored 
themselves  highly  competent  on  outcomes  to  be  those  with  earned  Ph.D.  degrees,  table  34 
shows  contrary  evidence  to  that  expectation.  On  the  average,  the  percentages  of 
respondents  with  Ph.D.s  scoring  themselves  highly  competent  on  outcome  variables 
appear  to  be  the  lowest  of  all  four  education  levels.  A  possible  explanation  for  this  may  be 
that  those  with  Ph.D.s  are  in  highly  specialized  areas  with  peripheral  jobs  that  do  not 
directly  relate  to  the  mainstream  Air  Force.  The  lowest  percentage  of  respondents  scoring 
themselves  highly  competent  continues  to  appear  in  the  Command  and  Leadership 
variable.  Again,  this  may  be  because  of  the  types  of  jobs  respondents  have  held  up  to  the 
point  in  time  when  they  attended  ACSC.  It  is  interesting,  however,  that  higher 
percentages  of  respondents  with  Master  Degrees  did  not  score  themselves  highly 
competent  on  outcome  variables  as  compared  to  other  education  levels,  especially  in  the 
Critical  Thinking  variable.  Perhaps  this  can  be  explained  by  looking  at  Master  Degree 
concentrations  (table  33)  which  shows  that  the  majority  of  respondents  fell  within  the 
Humanities  and  Social  Sciences.  Perhaps  respondents  in  these  areas  perceived  their 
critical  thinking  skills  as  lacking  because  of  their  choice  of  study  in  graduate  school.  Table 
34  seems  to  suggest,  however,  that  perceived  competence  on  outcome  variables,  on  the 
average,  goes  up  as  respondents  level  of  education  increases  from  Master  to  Master  Plus. 
It  also  suggests  that,  for  this  study  group,  obtaining  a  Ph.D.  does  not  necessarily  increase 
one’s  perception  of  competence  on  outcome  variables. 

Tables  summarizing  individual  items  (C-5  and  C-6)  show  similar  patterns  as  those 
mentioned  in  the  above  discussion.  The  Medical/Legal  Degree  Concentration  by  item 
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comparison  shows  many  high  percentages,  with  a  large  cluster  of  one-hundred  percents 
(100%)  in  the  Critical  Thinking  variable.  By  item  comparisons  across  the  other  four 
categories  seem  to  hold  steady  with  the  largest  percentages  of  high  competence  for  aU 
four  categories  appearing  in  the  Air/Space  Power  variable.  By  item  comparisons  of 
Education  Level  reveal  that  the  largest  percentages  of  respondents  scoring  themselves 
highly  competent  were  on  item  23,  “understanding  the  circumstances  of  war,”  rather  than 
item  31  (item  16  was  still  the  lowest).  An  interesting  observation  about  the  comparison  by 
item  on  Education  Level  is  the  low  scores  foimd  in  the  Ph.D.  column.  Yet,  relative  to  the 
other  categories,  respondents  with  Ph.D.s  scored  their  competencies  as  high  on  several 
items,  mostly  in  the  Critical  Thinking  variable.  Items  22-26  stand  out  in  the  Critical 
Thinking  variable  for  Ph.D.s  as  items  in  which  a  large  percentage  of  respondents  perceived 
high  competence.  These  items  deal  with  understanding  concepts  and  being  able  to  apply 
them  in  new  and  challenging  situations.  It  makes  sense  that  respondents  with  Ph.D.s 
would  perceive  themselves  as  highly  competent  in  this  skill  because  of  their  specialized 
training,  which  is  evidenced  by  the  data  in  table  C-6. 

Commission  Source  and  Outcome  Variables.  The  final  demographic  category  looks 
at  the  percentages  of  respondents  scoring  high  on  outcome  variables  by  commission 
source.  The  three  commission  source  categories  are  Reserve  Officer  Training  Corps 
(ROTC),  United  States  Air  Force  Academy  (USAFA),  and  Officer  Training  School 
(OTS).  Table  35  summarizes  the  results  of  the  data  analysis. 
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Table  35 : 

Percentages  of  Respondents  Scoring  High  (an  average  of  4  and  5)  on  Outcome 
Variables  by  Commission  Source. 


Rate  (%)  Among 

Outcome  Variable 

ROTC 
n  =  91 

USAFA 
n  =  44 

OTS 
n  =  78 

Joint  Campaign 

45%  (41) 

34%  (15) 

35  %  (27) 

Air/Space  Power 

64  %  (58) 

66  %  (29) 

64  %  (50) 

Command  &  Leadership 

33  %  (30) 

32  %  (14) 

27%  (21) 

Critical  Thinking 

42  %  (38) 

48  %  (21) 

45  %  (35) 

The  table  shows  that  the  largest  percentages  of  respondents  scoring  themselves  highly 
competent  on  outcome  variables  were  commissioned  through  ROTC.  Academy  graduates 
followed  closely  to  ROTC  graduates  in  the  percentages,  with  the  lowest  percentages  of 
respondents  scoring  themselves  as  highly  competent,  on  the  average,  occurring  in  the  OTS 
column.  The  largest  percentages,  on  the  average,  of  respondents  scoring  themselves 
highly  competent  were  in  the  Air/Space  Power  variable,  the  smallest  were  in  the 
Command  and  Leadership  variable.  Although  percentages  vary  somewhat  across 
commission  sources,  the  table  reveals  consistency  in  most  cross  tabulations.  The  most 
dramatic  difference  on  the  table  occurs  in  the  ROTC  column  with  the  Joint  Campaign 
variable.  At  45%,  this  figure  is  ten  percentage  points  higher  than  the  other  two  columns. 

It  is  mentioned  because,  in  a  table  that  shows  relative  consistency,  the  difference  stands 
out.  This  seems  to  suggest  that  respondents  who  were  commissioned  by  ROTC  were 
more  confident  in  their  competency  in  Joint  Campaign  than  those  who  were  commissioned 
by  the  other  two  sources.  Again,  this  may  be  because  of  the  types  of  jobs  held  by  officers 
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in  that  category  rather  than  the  source  of  commission.  Although  commissioning  source 
does  not  dictate  the  types  of  career  field  or  jobs  assigned  to  ofScers,  it  is  still  possible  that 
more  ROTC  graduates  have  held  joint  assignments  than  their  contemporaries. 

Item-by-item  examination  of  the  variables  across  commission  sources  revealed  that 
items  1 6  and  3 1,  as  in  earlier  analyses,  maintained  the  lowest  and  highest  percentages 
(Appendbc  C-7)  across  categories.  Generally,  most  of  the  items  appear  to  be  consistent 
with  low  and  high  percentages  holding  constant  across  the  three  commission  sources. 

Next  to  item  1 6,  the  lowest  percentage  of  respondents  scoring  their  competence  as  high 
was  on  item  10,  “understanding  and  applying  quality  force  tools  and  techniques.” 
Conversely,  next  to  item  31,  a  larger  percentage  of  respondents  scored  themselves  high 
on  item  12,  “understanding  standards  of  conduct  and  officership.”  Overall,  the 
commission  source  variable  appears  to  have  the  most  homogeneity  of  percentages  across 
columns  and  less  variance  than  the  other  demographic  variables  examined  in  the  data 
analysis.  This  suggests  that  commission  source  may  not  be  associated  with  students’ 
perceptions  of  competence  on  outcome  variables. 

Student  Expectations  and  Outcome  Variables.  Student  expectations  consisted  of  nine 
items  in  which  students  were  asked  to  check  all  that  apply.  Since  students  could  check 
more  than  one  item,  each  expectation  is  a  discrete  variable.  Tables  36  and  37  summarize 
the  results  fi'om  the  data  analysis.  The  tables  show  the  percentages  of  respondents  scoring 
high  on  outcome  variables  by  student  expectations  (for  exact  definitions  of  each 
expectation,  see  question  88  on  the  questionnaire,  Appendk  A). 
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Table  36 : 

Percentages  of  Respondents  Scoring  High  fan  average  of  4  and  5)  on  Outcome 


Rate  (%)  Among 

Outcome  Variable 

Exp  1 
n  =  191 

Exp  2 
n  =  190 

Exp  3 
n  =  196 

Exp  4 
n  =  36 

Joint  Campaign 

48% 

(92) 

45% 

(86) 

42% 

(82) 

36% 

(13) 

Air/Space  Power 

66% 

(126) 

68% 

(129) 

67% 

(132) 

67% 

(24) 

Command  &  Leadership 

30% 

(57) 

33% 

(63) 

33% 

(64) 

25% 

(9) 

Critical  Thinking 

45% 

(86) 

48% 

(91) 

47% 

(92) 

39% 

(14) 

Table  37: 

Percentages  of  Respondents  Scoring  High  tan  average  of  4  and  5)  on  Outcome  Variables 


bv  Student  Expectations 

'5-9). 

Rate  (%)  Among 

Outcome  Variable 

Exp  5 
n  =  104 

Exp  6 
n  =  42 

Exp  7 
n  =  6 

Exp  8 
n  =  143 

Exp  9 
n  =  26 

Joint  Campaign 

39% 

(41) 

48% 

(20) 

33% 

(2) 

45% 

(65) 

46% 

(12) 

Air/Space  Power 

65% 

(68) 

64% 

(27) 

50% 

(3) 

68% 

(98) 

77% 

(20) 

Command  &  Leadership 

30% 

(31) 

24% 

(10) 

33% 

(2) 

38% 

(55) 

35% 

(9) 

Critical  Thinking 

46% 

(48) 

45% 

(19) 

67% 

(4) 

49% 

(70) 

42% 

(11) 

As  can  be  seen  from  these  tables,  the  range  of  percentages  across  student  expectations 
are  small.  The  largest  percentages  of  respondents  scoring  high  on  outcome  variables 
occur  in  the  Air/Space  Power  variable,  with  the  highest  association  on  the  two  tables 
between  expectation  9  and  Air/Space  Power.  Expectation  9  provided  respondents  with  an 
opportunity  to  write  in  additional  expectations  that  did  not  appear  on  the  list.  Apparently, 
of  those  who  had  additional  expectations  (other  than  those  on  the  list),  a  larger  percentage 
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were  likely  to  see  themselves  as  highly  competent  in  Air/Space  Power.  While  it  is  obvious 
that  those  who  scored  themselves  as  highly  competent  in  Air/Space  Power  expected 
something  that  did  not  appear  on  the  list  of  expectations,  what  that  might  be  is  not 
altogether  clear.  The  descriptive  summary  of  expectations  (see  table  3  and  subsequent 
discussion)  revealed  that  the  comments  on  expectation  9  varied  but  the  majority  fell  in  the 
professional  arena.  The  least  percentage  of  respondents  scoring  themselves  highly 
competent  on  outcome  variables  was  in  expectation  6,  “I  would  spend  a  great  deal  of  time 
socializing”  in  the  Command  and  Leadership  variable.  This  suggests  that  those  who 
perceived  their  competence  as  high  on  Command  and  Leadership  expected  to  work  hard 
and  not  be  involved  in  a  great  deal  of  socializing  while  a  student  at  ACSC.  Overall,  tables 
36  and  37  show  very  little  association  between  outcome  variables  and  expectations, 
indicating  that  student  expectations  had  little  to  do  with  their  perceptions  about 
competency  in  outcome  variables. 

In  conclusion,  some  interesting  relationships  between  input  and  outcome  variables 
emerged  as  a  result  of  the  data  analysis.  Across  aU  demographic  characteristics  and 
student  expectations,  the  largest  percentages  of  respondents  scoring  high  on  outcome 
variables  were  in  the  Air/Space  Power  variable,  the  smallest  were  in  the  Command  and 
Leadership  variable.  This  consistency  across  input  variables  indicates  that,  regardless  of 
student  characteristics  and  expectations,  more  respondents  perceived  their  competency  as 
high  in  Air/Space  Power  and  the  least  number  perceived  their  competency  as  high  in 
Command  and  Leadership.  While  it  is  not  surprising  that  this  study  group  of  Air  Force 
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officers  would  score  themselves  as  being  highly  competent  in  Air/Space  Power,  it  is 
somewhat  surprising  that  Command  and  Leadership  consistently  appeared  as  the  least 
perceived  highly  competent  category.  This  may  come  from  the  fact  that,  as  newly 
promoted  Majors,  most  officers  have  not  yet  held  positions  that  require  them  to  command 
and/or  lead. 

Another  interesting  relationship  was  between  gender  and  outcome  variables.  Given  the 
four  outcome  variables  and  the  nature  of  the  military  environment,  one  might  expect  a 
higher  percentage  of  male  respondents  to  score  themselves  highly  competent  in  outcome 
variables  than  female  respondents.  Table  30  shows  evidence  to  the  contrary.  In  all 
outcome  variables,  except  Critical  Thinking,  a  larger  percentage  of  female  respondents 
scored  their  competency  as  high  than  did  male  respondents.  This  table  suggests  that 
female  respondents  were  more  confident  in  their  competency  level  on  three  of  the  four 
outcome  variables  than  male  respondents  Similarly,  table  3 1  shows  that  African-American 
respondents,  on  the  average,  were  more  confident  in  their  competencies  than  those  in  the 
other  race  categories. 

While  one  might  assume  that  officers  in  the  Operations  career  field  (pilots,  navigators, 
missileers,  etc.)  and  those  who  have  Ph.D.s  would  score  themselves  as  highly  competent 
in  outcome  variables,  tables  32  and  34  show  that  this  was  not  the  case.  On  all  outcome 
variables,  except  Air/Space  Power,  the  percentages  of  officers  in  the  Operations  career 
field  scoring  themselves  as  highly  competent  were  less  than  those  in  other  career  fields. 
Similarly,  table  34  suggests  that  obtaining  a  Ph.D.  did  not  necessarily  increase  students’ 
perceptions  of  competence  on  outcome  variables. 
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Finally,  the  data  analysis  on  two  input  variables,  commission  source  and  student 
expectations,  revealed  that  the  percentages  of  respondents  scoring  themselves  as  highly 
competent  on  outcome  variables  did  not  vary  much  across  categories.  This  fact  seems  to 
suggest  that  there  was  very  little  association  between  commission  source  and  outcome 
variables  and  student  expectations  and  outcome  variables.  In  other  words,  respondents’ 
perceived  level  of  competence  on  outcome  variables  was  not  very  affected  by  their  source 
of  commission  or  their  expectations. 


The  Relationship  Between  Environmental  Variables  and  Outcome  Variables 

Question  #5  from  chapters  one  and  three  of  this  dissertation  dealt  with  the  relationship 
between  environmental  variables  and  outcome  variables.  The  question  was:  “Was  there  a 
relationship  between  teaching  methods  and  program  activities  (environment)  and  student 
outcomes?”  Environmental  variables  consisted  of  six  categories  (teaching  methods, 
technology,  curriculum,  research,  faculty,  and  grading).  Items  50-74  on  the  questionnaire 
comprised  the  environmental  variables  used  for  the  data  analysis.  Items  75-81  were  meant 
to  be  descriptive  items  only  and  were  not  included  in  the  correlations  or  the  regression 
equations.  Therefore,  questionneiire  items  corresponded  to  environmental  variables  in  the 
following  manner: 


Teaching  Methods 

Technology 

Curriculum 

Research 

Faculty 

Grading 


Items  50-53 
Items  54-58,  and  74 
Items  59-62,  and  67 
Items  63,  68,  and  69 
Items  64-65,  and  70 
Items  66,  and  71-73 
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As  a  reminder,  the  four  outcome  variables  were  Joint  Campaign  (items  35-45),  Air/Space 
Power  (items  46-49),  Command  and  Leadership  (items  5-16),  and  Critical  Thinking  (items 
1 7-34).  As  specified  in  chapters  one  and  three,  data  on  correlations  between  environment 
and  outcome  variables  were  reported  using  Pearson  Correlation  Coefficients  (Pearson 
“r”s).  The  sfac  environmental  variables  and  four  outcome  variables  were  “collapsed”  by 
adding  together  the  items  which  made  up  each  variable.  For  example,  the  Teaching 
Method  variable  was  collapsed  by  adding  together  items  50-53  fi-om  the  questionnaire. 
Correlations  between  both  collapsed  variables  and  individual  items  were  reported.  Data 
on  individual  items  were  reported  in  table  format  and  are  found  in  the  Appendk 
(Appendfac  D).  Correlations  on  collapsed  values  may  be  larger  than  correlations  for 
individual  items  because  there  was  a  wider  range  of  scores  on  those  variables.  When  there 
are  a  wider  range  of  scores,  the  variance  is  greater,  thereby  increasing  the  size  of  the 
correlations.  Because  of  the  number  of  variables  involved  in  computing  the  correlations, 
the  potential  for  making  a  Type  I  error  increases.  Therefore,  the  significance  level  was  set 
at  .01,  which  limited  the  probability  of  a  Type  I  error  to  one  in  a  hundred.  By  setting  the 
significance  level  at  .01,  the  reader  can  be  ninety-nine  percent  (99  %)  certain  that  there 
was  a  relationship  between  the  variables  where  one  was  indicated. 

Table  38  shows  the  relationships  between  the  collapsed  environmental  and  outcome 
variables.  As  the  table  indicates,  there  were  moderate  positive  correlations  between 
variables  in  most  cases  which  were  significant  at  the  .01  level.  The  largest  relationship  in 
this  table  was  between  Curriculum  and  Joint  Campaign  (r=.46).  This  relationship  shows 
that  those  who  rated  the  quality  of  the  curriculum  as  high  were  more  likely  to  rate  their 
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competence  level  in  Joint  Campaign  as  high  as  well.  With  a  correlation  coefficient  of  .44, 
Teaching  Methods  and  Joint  Campaign  were  also  highly  correlated,  relative  to  the  other 
variables.  The  lowest  relationship  between  two  variables  that  was  statistically  significant 
at  the  .01  level  was  Research  and  Critical  Thinking.  Although  statistically  significant,  the 
low  correlation  coefficient  suggests  that  this  relationship  was  relatively  weak.  Finally, 
according  to  the  data  presented  in  the  table,  one  notes  that  the  perceived  quality  of 
research  appears  to  be  least  associated  with  the  outcome  variables.  The  most  associated 
column  with  outcomes  appears  to  be  Curriculum. 

Table  38: 

Correlations  Between  Environmental  and  Outcome  Variables 


Variable 

Teaching 

Method 

Research 

Joint  Campaign 

.46* 

.25* 

.36* 

.26* 

Air/Space  Power 

.36* 

.25* 

.41* 

.10 

.30* 

.25* 

Command  &  Leadership 

.32* 

.25* 

.39* 

.25* 

.35* 

.24* 

Critical  Thinking 

.39* 

.23* 

.41* 

.13* 

.31* 

.22* 

*  Significant  at  .01  Level 

Appendix  D  contains  the  tables  in  which  correlations  between  all  six  environmental 
variables  and  the  four  outcome  variables  are  summarized  by  item.  The  summaries  reveal 
significant  relationships  between  items.  The  highest  relationship  on  table  D-1,  was 
between  items  53  and  39.  At  r=.36,  this  correlation  shows  a  moderate  positive  correlation 
between  items  53,  “colloquia  style  seminars,”  and  39,  “understanding  the  roles,  functions, 
capabilities  and  limitations  of  the  U.S.  military  forces  that  affect  joint  and  combined 
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operations.”  In  the  Air/Space  Power  variable  all  items  reached  significance,  but  in  the 
Command  and  Leadership  variable,  many  did  not,  especially  with  item  51,  “lectures  by 
guest  lecturers.”  The  lowest  correlation  coefficient  in  the  Command  and  Leadership 
variable  (not  significant)  was  a  -.01  between  items  51  and  hern  8,  “applying  principles  of 
oral  and  written  communication  in  a  variety  of  situations.”  This  shows  an  extremely  low 
inverse  relationship  in  which  the  two  variables  tend  to  go  in  opposite  directions.  Overall, 
table  D-1  shows  a  significant  correlation  between  Teaching  Methods  and  outcome 
variables  except  for  Command  and  Leadership  where  items  50  and  5 1  (items  dealing  with 
lectures)  show  very  little  correlation  overall.  This  infers  that  the  perceived  quality  of 
lectures  at  ACSC  were  least  associated  with  the  Command  and  Leadership  variable.  It 
may  be  that  those  with  perceived  high  competence  on  Command  and  Leadership  did  not 
prefer  the  lecture  method  because  command  and  leadership  skills  require  a  high  level  of 
involvement  with  others,  therefore,  these  individuals  may  be  more  accustomed  to  active 
rather  than  passive  learning. 

Table  D-2  in  the  Appendix  summarizes  correlations  between  Technology  and  outcome 
variables  by  item.  Overall,  the  size  of  the  correlations  between  Technology  items  and 
outcome  variable  items  is  not  very  large.  The  highest  Pearson  “r”  was  .28  between  items 
56,  “  ‘Toolbook’  programs  to  supplement  readings,”  and  10,  “understanding  and  applying 
quality  force  tools  and  techniques.”  An  important  observation  in  table  D-2  is  the  number 
of  low  correlations  appearing  in  both  the  Command  and  Leadership  and  Critical  Thinking 
cells.  While  some  significant  correlations  appear,  the  number  of  low  correlations  indicate 
that,  in  general.  Technology  has  a  very  low  correlation  vrith  Command  and 
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Leadership  and  Critical  Thinking.  Perhaps  those  who  had  a  tendency  to  rate  the  quality  of 
Technology  high  were  in  technology-oriented  jobs  that  did  not  require  command  and 
leadership  or  critical  thinking  skills.  This  would  account  for  the  low  correlations  between 
Technology  and  these  two  outcome  variables. 

Higher  correlations  are  found  in  table  D-3.  There  were  a  number  of  significant 
correlations  in  all  four  outcome  variables.  The  highest  significant  correlation  at  r  =  .41 
occurred  in  the  Air/Space  Power  variable  between  item  61  and  items  46  and  48. 
Curriculum  item  61,  “systems  approach  to  problem  solving,”  was  moderately  correlated  to 
both  item  46,  “understanding  the  role  and  impact  of  technology  on  air  campaign 
planning,”  and  item  49,  “understanding  and  analyzing  the  impact  of  contextual  and 
operational  art  elements  on  military  campaigns.”  The  fewest  significant  correlations  were 
between  items  59  and  60  and  Command  and  Leadership.  Item  59,  “assigned  books,”  and 
item  60,  “curriculum  flow,”  appear  to  be  least  associated  with  Command  and  Leadership. 
Again,  perhaps  those  who  scored  their  competence  as  high  in  Command  and  Leadership, 
did  not  like  the  assigned  books  or  the  curriculimi  flow  because  they  preferred  other  types 
of  more  active  learning.  For  example,  those  with  a  high  degree  of  perceived  Command 
and  Leadership  skills  may  have  preferred  more  role  playing  activities.  Overall,  the  number 
of  significant  correlations  appearing  in  table  D-3  seem  to  infer  that  there  was  a  significant 
relationship  between  Curriculum  and  the  four  outcome  variables.  Although  there  were  a 
number  of  items  on  which  significance  was  not  reached,  for  the  most  part,  respondents 
who  rated  the  quality  of  Curriculum  as  high  tended  to  rate  their  competence  level  on 
outcome  variables  as  high  as  well.  This  may  occur  because  respondents  were  Air  Force 
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officers  who  have  been  in  the  Air  Force  for  about  twelve  years.  ACSC  was  probably  not 
their  first  experience  with  professional  military  education  and  they  were  used  to  the  way 
Air  Force  PME  generally  operates—the  curriculum  flow,  the  systems  approach,  and  the 
like. 

Table  D-4  summarizes  results  of  the  data  analysis  between  Research  and  outcome 
variables  by  item.  Although  there  were  several  significant  correlations  on  the  table,  overall 
the  correlations  were  extremely  low,  especially  within  the  Air/Space  Power  and  Critical 
Thinking  variables.  The  highest  relationship  appeared  in  the  Joint  Campaign  variable 
between  item  69,  “the  extent  to  which  students  received  adequate  support  with  their 
research  projects,”  and  item  36,  “understanding  the  structure  of  a  joint  organization.” 

Item  1 7,  “challenging  assumptions,”  shows  a  low  inverse  relationship  with  all  three 
Research  variables.  The  data  in  table  D-4  infer  that  Research  has,  with  a  few  exceptions, 
little  to  no  relationship  with  outcome  variables.  This  corroborates  the  data  in  table  38 
which  showed  that  the  perceived  quality  of  research  was,  when  compared  to  other 
environmental  variables,  least  associated  with  the  outcome  variables. 

Correlations  between  feculty  and  outcome  variables  are  displayed  in  table  D-5.  There 
were  many  significant  correlations  especially  between  items  64,  “military  faculty 
members,”  and  65,”  civilian  faculty  members,”  and  outcome  variables.  The  highest 
Pearson  “r”  was  .33,  appearing  three  times  in  the  military  faculty  members  (item  64) 
column.  An  interesting  observation  in  table  D-4  was  the  low  correlations  between  item 
70,  “the  extent  to  which  the  faculty  should  consist  of  more  Ph.D.s”  and  outcome  variables. 
With  the  exception  of  items  9,  25,  and  29,  which  achieved  significance,  there  appears  to  be 
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no  relationship  between  student  perceptions  of  the  number  of  Ph.D.s  at  ACSC  and  their 
perceived  competence  on  outcome  variables.  These  data  seem  to  indicate  that 
respondents  perceived  that  increasing  the  number  of  Ph.D.s  on  the  faculty  would  not 
necessarily  improve  the  quality  of  the  faculty. 

The  final  table  in  Appendbc  D,  table  D-6,  lists  correlations  between  Grading  and 
outcome  variables.  According  to  the  data  in  table  D-6,  there  were  a  few  significant 
correlations,  especially  in  the  Joint  Campaign  and  Air/Space  Power  variable.  There  are 
fewer  significant  correlations  in  the  Command  and  Leadership  and  Critical  Thinking 
variables.  However,  the  highest  Pearson  “r”  at  .3 1  occurs  in  the  Critical  Thinking  variable 
between  item  72,  “the  extent  to  which  students  knew  about  and  understood  the  reclama 
process,”  and  item  20,  “understanding  and  implementing  change  in  a  military 
environment.”  Apparently,  respondents  who  rated  their  competence  on  “change  in  a 
military  environment”  high  were  also  more  inclined  to  rate  the  quality  of  the  “reclama 
process”(ACSC’s  system  of  disputing  grades)  as  high  as  well.  Perhaps  individuals  who 
were  more  receptive  to  change  were  more  accepting  of  the  way  in  which  grades  could  be 
disputed  at  ACSC.  Others,  as  evidenced  by  table  D-6,  did  not  have  the  same  perception. 
Although  there  were  a  number  of  other  significant  correlations  appearing  on  the  table, 
overall  the  relationship  between  Grading  and  outcome  variables  was  low.  The  most 
highly  correlated  column  with  outcome  variables  was  item  71,  “the  extent  to  which 
students  received  feedback  on  oral  and  written  work,”  as  is  evidenced  by  the  number  of 
correlations  reaching  significance  in  that  column,  especially  in  the  Joint  Campaign  variable. 
It  is  possible  that  many  respondents  who  rated  their  competence  high  in  Joint  Campaign 
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have  held  joint  assignments  (those  in  conjunction  with  other  services)  in  the  past.  Perhaps 
this  distinction  would  make  those  individuals  more  inclined  to  rate  the  quality  of  feedback 
for  their  work  as  high.  In  other  words,  the  nature  of  joint  assignments  may  be  the 
distinction  that  created  the  perception  that  feedback  on  oral  and  written  work  at  ACSC 
was  of  high  quality.  Conversely,  item  66,  “grading  procedures”  shows  the  least 
relationship  with  all  four  outcome  variables.  These  data  suggest  that  students  did  not  rate 
the  grading  procedures  at  ACSC  as  being  of  high  quality  regardless  of  their  perceptions  of 
their  competence  level  on  outcomes. 

In  conclusion,  results  of  the  data  analysis  to  show  the  relationships  between 
environmental  and  outcome  variables  were  reported  m  table  38.  These  values  reflected 
“collapsed”  values  on  both  environmental  and  outcome  variables.  Item-by-item 
relationships  were  reported  in  tables  D-1  through  D-6  at  Appendix  D.  Table  38  revealed 
that,  on  the  average,  environmental  variables  were  associated  with  outcomes  in  the 
following  manner  (from  most  to  least):  Curriculum,  Teaching  Methods,  Faculty, 
Technology,  Grading,  and  Research.  The  largest  relationship  was  between  Curriculum 
and  the  Joint  Campaign  outcome  variable  (r=.46),  and  the  smallest  statistically  significant 
relationship  was  between  Research  and  Critical  Thinking  (r=  .13).  Interestingly  enough, 
the  relationship  between  Research  and  Air/Space  Power  did  not  reach  significance  at  the 
.01  level.  This  indicates  that  there  was  no  relationship  between  Research  and  the 
Air/Space  Power  outcome  variable. 
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Tables  D-1  through  D-6  at  Appendk  D  corroborate  the  data  in  table  38  but  gives  more 
details  about  item-by-item  relationships.  Overall,  these  data  show  that,  the  environmental 
variables  were  least  associated  with  the  Command  and  Leadership  and  Critical  Thinking 
outcome  variables.  Although  the  relationship  between  some  items  did  reach  significance, 
generally  speaking,  these  two  outcome  variables  show  little  to  no  relationship  with 
environmental  variables.  One  possible  explanation  why  this  occurred  is  that  Air  Force 
officers,  as  a  group,  do  not  perceive  their  competence  in  Command  and  Leadership  or 
Critical  Thinking  as  being  high  because  of  the  types  of  jobs  they  have  held  up  to  this  point 
in  their  careers. 


Predictive  Analysis 

The  specific  predictive  question  posed  in  chapters  one  and  three  of  this  dissertation 
was:  “Holding  the  input  variables  (demographics  and  student  expectations)  constant,  was 
there  a  relationship  between  environmental  variables  (teaching  methods  and  program 
activities)  and  student  outcomes?”  As  stated  earlier,  multiple  regression  was  the  statistical 
procedure  used  to  determine  the  effect  of  environmental  variables  on  outcome  variables 
while  “controlling”  for  the  effects  of  input  variables.  The  researcher  expected  that 
demographics  and  student  expectations  (inputs)  would  correlate  with  outcome  measures. 
As  a  result,  any  observed  correlation  between  environmental  variables  (teaching  methods 
and  program  activities)  and  students’  perceptions  of  competence  on  outcomes  may  have 
occurred  because  of  student  input  characteristics.  To  obtain  a  less  biased  estimate  of  the 
effects  of  teaching  methods  and  program  activities  at  ACSC  on  students’  perceptions  of 


134 


competence  on  outcome  measures,  it  was  necessary  to  hold  the  input  variables  constant. 
Therefore,  the  first  block  of  variables  in  the  regression  consisted  of  demographics  and 
student  expectations.  By  entering  the  input  variables  in  the  regression  first,  the  researcher 
was  able  to  tell  if  the  teaching  methods  and  program  activities  at  ACSC  contributed 
anything  to  the  prediction  of  student  perceptions  of  competence  on  outcome  variables 
beyond  what  could  be  predicted  fi’om  student  input  characteristics  (Astin,  1993,  pg.  285). 
There  are  four  outcome  variables— Joint  Campaign,  Air/Space  Power,  Command  and 
Leadership,  and  Critical  Thinking.  Results  of  the  regression  analyses  on  all  four  outcome 
variables  were  summarized  and  reported. 

The  researcher  used  the  forward  selection  method  in  SPSS  to  enter  variables. 
Independent  variables  were  entered  as  two  “blocks,”  input  variables  as  block  one  and 
environmental  variables  as  block  two.  The  researcher  established  the  criterion  for 
inclusion  in  the  regression  equation  at  a  confidence  level  of  .05  (F>3.85).  SPSS  entered 
the  variables  fi:om  the  first  block  one  at  a  time.  The  first  variable  considered  for  entry  was 
the  one  with  the  largest  positive  or  negative  correlation  with  the  dependent  variable.  If  the 
first  variable  met  the  criterion  for  inclusion,  the  procedure  continued.  Once  the  first 
variable  was  entered,  the  program  then  moved  on  to  look  at  the  partial  correlations 
between  the  dependent  variable  and  each  of  the  independent  variables  not  yet  entered  into 
the  equation.  The  partial  correlation  was  the  correlation  between  an  independent  variable 
and  the  dependent  variable  that  was  adjusted  for  the  variables  already  in  the  equation.  The 
variable  with  the  largest  partial  correlation  was  the  next  candidate  for  entry  into  the 
equation.  This  procedure  was  repeated  until  there  were  no  other  variables  that  met  the 
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entry  criterion  (Norusis,  1993,  pg.  347).  When  all  of  the  predictive  power  of  the  input 
variables  were  exhausted,  the  program  then  moved  to  the  variables  in  the  second 
(environmental)  block.  The  procedure  continued  with  the  second  block  of  variables  in  the 
same  manner  as  the  first  block.  When  all  of  the  predictive  power  of  both  blocks  of 
variables  were  exhausted,  the  program  terminated  (Astin,  1993,  pg.  285). 

Predictions  on  Joint  Campaign  Planning  Process 

Table  39  shows  the  results  of  the  regression  procedure  on  the  outcome  variable.  Joint 
Campaign  Planning  Process.  The  first  column  shows  the  step  number  in  the  regression. 
The  second  column  lists  the  variable  entered  at  each  step.  The  Multiple  R  column  lists  the 
multiple  correlation  coefficient  at  each  step  in  the  equation.  The  Multiple  R  is  the  simple 
correlation  between  the  outcome  variable  and  the  estimate  of  that  outcome  derived  fi'om 
the  regression  equation  at  that  step  in  the  analysis  (Astin,  1993,  pg.  285).  The  next 
column  shows  the  simple  Pearson  “r”  between  the  entering  variable  and  the  outcome 
variable.  R-squared  is  the  square  of  the  Multiple  R,  but  it  also  represents  “the  proportion 
of  the  variation  in  the  dependent  variable  ‘explained’  by  the  model”  (Norusis,  pg.  320). 

By  looking  at  the  change  in  R-squared,  the  researcher  can  determine  to  what  extent 
variation  was  explained  by  variables  above  what  was  explained  by  the  other  variables 
already  in  the  equation. 
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Table  39: 

Predicting  Student  Perceptions  of  Competence  on  Joint  Campaign  Planning  Process. 


Step 

Variable 

Multiple  R 

Simple  r 

R-squared 

R-squared 

Change 

Input 

1 

Age  2 

.18 

.17 

.03 

.03 

2 

Exp  8 

.27 

.18 

.08 

.05 

Environment 

3 

Curriculum 

.48 

.46 

.23 

.15 

4 

Teach  Methods 

.51 

.44 

.26 

.03 

This  table  shows  that  two  input  variables,  Age  2,  the  36-39  age  group,  and  Expectation 
8,  “I  would  be  academically  challenged  while  a  student  at  ACSC,”  entered  the  equation  at 
steps  one  and  two.  This  means  that  those  who  were  in  the  age  group,  36-39,  and  who 
expected  to  be  academically  challenged  at  ACSC,  were  more  likefy  to  perceive  greater 
competence  on  the  Joint  Campaign  outcome  variable.  After  the  student  input 
characteristics  were  controlled,  two  environmental  variables  entered  the  regression 
equation  in  the  second  stage  (steps  3  and  4).  These  data  suggest  that  the  curriculum  and 
teaching  methods  at  ACSC  affected  students’  perceptions  of  competence  on  Joint 
Campaign  after  student  inputs  were  controlled.  Eight  percent  (8  %)  of  the  variance  was 
explained  by  the  inputs,  and  Curriculum  explained  fifteen  (15  %)  of  the  variance  in  the 
equation  above  what  was  explained  by  the  inputs.  When  Teaching  Methods  was  added  to 
the  equation  at  step  four,  it  only  added  three  percent  (3  %)  more  to  the  equation.  This 
occurred  because  most  of  the  variation  was  explained  by  Curriculum  before  Teaching 
Methods  was  added.  This  suggests  that,  after  controlling  for  input  variables,  students 
who  rated  Curriculum  of  high  quality  were  fifteen  percent  (15  %)  more  likely  to  score 
themselves  as  highly  competent  on  Joint  Campaign.  Teaching  Methods  added  a  small 
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amount  to  the  equation  meaning  that,  after  Curriculum,  those  who  rated  Teaching 
Methods  of  high  quality,  were  three  percent  (3  %)  more  likely  to  score  themselves  as 
highly  competent  in  Joint  Campaign. 

Four  environmental  variables  (Faculty,  Technology,  Grading,  and  Research)  did  not 
even  enter  the  equation.  Table  40  shows  what  happened  when  these  four  variables  were 
forced  into  the  equation. 


Table  40: 


Predicting  Student  Perceptions  of  Competence  on  Joint  Campaign  Planning  Process:  Forcing  in 
Environmental  Variables  Not  In  Equation. _ 


Step 

Variable 

Multiple  R 

Simple  r 

R'Squared 

R-squared 

Change 

5 

Faculty 

.53 

.36 

.28 

.02 

6 

Technology 

.53 

.32 

.28 

.00 

7 

Grading 

.53 

.26 

.28 

.00 

8 

Research 

.53 

.25 

.28 

.00 

This  table  indicates  that  Faculty  added  a  small  amount  (2  %)  to  the  equation  but  that 
Technology  and  Grading  did  not  add  anything  to  the  prediction  of  students’  perceptions  of 
competence  on  the  Joint  Campaign  Planning  Process  when  variables  were  forced  into  the 
equation  after  inputs  and  Curriculum  and  Teaching  Methods  were  already  entered. 

Although  tables  39  and  40  show  the  results  of  the  data  analysis  when  all  environmental 
variables  are  entered  as  a  block,  they  do  not  show  what  would  happen  if  each 
environmental  variable  entered  the  equation  individually  after  inputs  were  controlled. 

Table  41  shows  the  predictive  power  of  each  environmental  variable  entered  individually 
after  input  variables  were  entered  into  the  equation.  The  R-squared  change  values  in  the 
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last  column  indicate  the  amount  of  variance  each  environmental  variable  added  individually 
to  the  equation  after  inputs. 

Table  41: 


Predicting  Student  Perceptions  of  Competence  on  Joint  Campaign  Planning  Process: 
Environmental  Variables  Entered  Separately  After  Inputs. _ 


Variable 

Multiple  R 

R-$quared 

R-squared 

Change 

Inputs  after 

21 

.08 

.05 

step  2 

Curriculum 

.48 

.23 

.15 

Teach  Methods 

.45 

.20 

.12 

Faculty 

.41 

.17 

.09 

Technology 

.40 

.16 

.08 

Grading 

.38 

.14 

,06 

Research 

.36 

.13 

.05 

These  data  show  that  the  same  two  variables,  Curriculum  and  Teaching  Methods, 
accounted  for  the  most  variation  after  inputs  even  when  they  are  entered  in  the  equation 
individually.  Notice  that  Curriculum  added  the  same  amount  of  variance  (15  %)  above 
inputs  in  both  equations  (tables  39  and  41).  However,  when  Teaching  Methods  entered 
the  equation  individually  after  inputs,  it  added  twelve  percent  (12  %)  above  what  was 
accoxmted  for  by  the  input  variables.  This  was  much  higher  than  the  three  percent  (3  %) 
that  Teaching  Methods  added  to  the  equation  when  it  had  to  share  predictive  power  with 
Curriculum  (table  39).  Similarly,  Faculty  did  not  enter  the  equation  on  table  39  based  on 
the  established  criterion,  but  it  accounted  for  nine  percent  (9  %)  of  the  variation  above 
input  variables  when  it  did  not  have  to  share  predictive  power.  This  indicates  that,  in 
relation  to  the  other  environmental  variables,  it  was  still  moderately  important  in 
predicting  students’  perceptions  of  competence.  The  other  three  environmental  variables 
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(Technology,  Grading,  and  Research)  also  accounted  for  a  moderate  amount  of  the 
variation.  This  indicates  that  even  though  these  variables  did  not  enter  the  equation  when 
they  had  to  share  with  the  other  variables,  but  when  taken  individually,  they  still  show 
some  predictive  power  after  inputs  were  controlled.  The  reason  for  this  is  that 
independent  variables  tend  to  be  correlated  with  each  other,  which  is  called 
“multicolinearity.”  As  more  variables  were  added  to  the  equation,  they  shared  increasing 
amounts  of  variance  with  each  other,  attributing  most  of  the  shared  variance  to  the 
variable  that  entered  the  equation  first. 


Predictions  on  Air/Space  Power  Applications 

Table  42  summarizes  the  results  of  the  regression  analysis  for  Air/Space  Power 
Applications.  It  shows  that  three  input  variables  entered  the  equation:  The  Medical 
/Legal  career  field.  Age  2  (36-39  age  group),  and  Expectation  8,  “I  would  be  academically 
challenged  while  a  student  at  ACSC.” 


Table  42: 


Predicting  Student  Perceptions  of  Competence  on  Air/Soace  Power  Applications. 


Step 

Variable 

Multiple  R 

Simple  r 

R-squared 

R-squared 

Change 

Input 

1 

Med/Legal 

.18 

.18 

.03 

.03 

2 

Age  2 

.25 

.17 

.06 

.03 

3 

.29 

.13 

.08 

.02 

Environment 

4 

Curriculum 

.46 

.41 

.21 

.13 

5 

Teach  Methods 

.48 

.36 

.23 

.02 

Table  42  reveals  similar  data  as  that  listed  in  table  39,  with  Age  2  and  Expectation  8 
entering  the  equation.  However,  the  Medical/Legal  career  field  entered  the  equation  at 
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Step  one,  indicating  that  it  had  the  most  predictive  power  on  students’  perceptions  of 
competence  on  the  Air/Space  Power  Applications  (3  %).  Age  2  added  three  percent  (3 
%)  above  the  Medical/Legal  career  field  and  Expectation  8  added  two  percent  (2  %)  even 
after  Medical/Legal  and  Age  2  entered  the  equation.  A  total  of  eight  percent  (8  %)  of  the 
variance  was  attributed  to  input  variables.  This  means  that  respondents  who  were  in  the 
Medical/Legal  career  field,  36-39  years  old,  and  who  expected  to  be  academically 
challenged  were  more  likely  to  score  themselves  highly  competent  on  Air/Space  Power 
Applications.  The  environmental  variables  that  entered  the  equation  after  controlling  for 
inputs  were,  once  again.  Curriculum  and  Teaching  Methods.  However,  Curriculum 
explained  less  variation  in  the  Air/Space  Power  outcome  variable  than  it  did  in  the  Joint 
Campaign  variable  (.15  versus  .13).  This  can  be  attributed  to  the  inclusion  of  the 
Medical/Legal  career  field  in  table  42.  Multicolinearity  between  the  independent  variables 
reduced  the  amount  of  variation  explained  by  the  environmental  variables  after  inputs  were 
added  to  the  equation.  Curriculum  still  accounted  for  thirteen  percent  (13  %)  of  the 
variation  above  that  explained  by  input  variables.  Teaching  Methods,  when  added  to  the 
equation,  only  accoimted  for  two  percent  (2  %)  of  the  variation.  Even  though  Teaching 
Methods  entered  the  equation,  the  amount  of  variation  it  explained  was  small,  indicating 
that  it  had  very  little  predictive  power  when  it  accompanied  Curriculum.  Again,  this  can 
be  explained  by  the  multicolinearity  between  environmental  variables. 

Table  43  shows  what  happened  when  environmental  variables  that  did  not  enter  the 
equation  were  forced  into  the  regression  after  step  5.  These  data  reveal  that  the  reason 
these  environmental  variables  did  not  enter  the  equation  was  they  did  not  add  anything  to 
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the  equation  over  what  was  accounted  for  by  Curriculum  and  Teaching  Methods.  By  the 
time  the  four  remaining  environmental  variables  entered  the  equation,  there  was  very  little 
shared  variance  remaining  for  them,  due  to  their  intercorrelations  with  the  other  variables. 


Table  43: 


Predicting  Student  Perceptions  of  Competence  on  Air/Soace  Power  Applications:  Forcing  in 
Environmental 
Variables  Not  In  Equation. 


Step 

Variable 

Multiple  R 

Simple  r 

R-squared 

R-squared 

Change 

6 

Faculty 

.49 

.30 

.24 

.01 

7 

Technology 

.49 

.25 

.24 

.00 

8 

Grading 

.49 

.25 

.24 

.00 

9 

Research 

.50 

.10 

.25 

,01 

Notice  that  Research,  which  was  forced  into  the  equation  at  step  9,  actually  accounted 
for  one  percent  (1  %)  of  the  variation  when  all  other  variables  were  already  in  the 
equation.  This  was  a  very  small  amount  but  it  stood  out  because  of  the  low  simple 
correlation  Research  had  with  the  outcome  variable  (r=.10  which  was  not  significant)  and 
because  the  variables  entering  the  equation  at  steps  7  and  8  did  not  account  for  any  change 
in  variation.  This  can  be  explained  by  looking  at  the  partial  regression  coefiBcients  of  the 

variable  is  contingent  upon  the  other  variables  in  the  equation  (Norusis,  pg.  341).  The 
partial  regression  coefficients  of  the  variables  entered  in  the  equation  fi-om  tables  42  and 
43  are  as  follows: 


Med/Legal  2.0452 1 6 

Age  2  .788672 

Exp  8  .049422 

Curriculum  .202972 
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Teach  Methods 

.152686 

Faculty 

.072617 

Technology 

.043424 

Grading 

.067152 

Research 

-.143189 

These  figures  indicate  that  after  statistical  adjustment  for  the  other  independent  variables 
in  the  equation,  Research  negatively  affected  students’  perceptions  of  competence  on 
Air/Space  Power  Applications  to  a  greater  degree  than  Faculty,  Technology  or  Grading 
affected  it  positively.  In  other  words,  respondents  who  rated  the  quality  of  Research  as 
low  were  those  who  were  most  likely  to  score  themselves  highly  competent  in  Air/Space 
Power  Applications. 

When  the  environmental  variables  were  entered  into  the  equation  individually  after  the 
input  variables  were  controlled  and  their  R-squared  values  compared  to  that  of  inputs  after 
step  3,  the  amount  of  variance  explained  by  individual  variables  are  shown.  These  data  are 
summarized  in  table  44. 


Table  44: 

Predicting  Student  Perceptions  of  Competence  on  Air/Space  Power  Applications: 


Environmental  Variables  Entered  Separately  After  Inputs. 


Variable 

Multiple  R 

R-squared 

R'Squared 

Change 

Inputs  after 

.29 

.08 

.08 

Step  3 

Curriculum 

.46 

.21 

.13 

Teach  Methods 

.42 

.18 

.10 

Faculty 

.37 

.14 

.06 

Technology 

.38 

.14 

.06 

Grading 

.38 

.14 

.06 

Research 

.30 

.09 

.01 
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These  data  corroborate  data  from  tables  42  and  43,  showing  Curriculum  and  Teaching 
Methods  as  accounting  for  the  most  variance.  Three  variables,  Faculty,  Technology,  and 
Grading,  appeared  to  account  for  the  same  amount  of  variance  when  entered  individually 
after  inputs.  Research,  as  suspected,  accounted  for  the  least  amount  of  variance  showing 
that  it  accounted  for  only  one  percent  (1  %)  above  what  was  accounted  for  by  the  inputs. 


Predictions  on  Command  and  Leadership 

Table  45  summarizes  the  regression  data  analysis  on  the  students’  perceptions  of 
competence  on  the  Command  and  Leadership  variable.  These  data  reveal  that  only  one 
input  variable  (Expectation  8)  entered  the  equation.  After  the  input  variables  were 
controlled.  Curriculum  and  Faculty  entered  the  equation  at  steps  2  and  3. 


Table  45: 

Predicting  Student  Perceptions  of  Competence  on  Command  and  Leadership. 


Step 

Variable 

Multiple  R 

Simple  r 

R-squared 

R-squared 

Change 

Input 

1 

Exp  8 

.19 

.21 

.04 

.03 

Environment 

2 

Curriculum 

.38 

.39 

.14 

.10 

3 

Faculty 

.42 

.35 

.18 

.04 

According  to  the  data  in  this  table,  once  input  variables  were  controlled.  Curriculum 
accoimted  for  ten  percent  (10  %)  of  the  explained  variance  above  what  was  accounted  for 
by  the  inputs.  With  Curriculum  in  the  equation.  Faculty  came  in  at  step  3,  accoimting  for 
only  four  percent  (4  %)  of  the  variance  above  that  of  Curriculum.  This  suggests  that 
Curriculum,  as  in  the  other  regression  analyses,  had  the  greatest  predictive  power  on 
students’  perceptions  of  competence  on  Command  and  Leadership  after  controlling  for 
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input  variables.  Again,  because  of  multicolinearity,  most  of  the  variance  was  attributed  to 
Curriculum,  leaving  little  left  over  for  the  other  variables  to  share. 

In  order  to  see  how  much  variance  was  explained  by  the  remaining  environmental 
variables  which  did  not  enter  the  equation,  these  four  variables  (Teaching  Methods, 
Technology,  Research,  and  Grading)  were  forced  into  the  equation  at  steps  4-7  in  the 
equation.  Table  46  summarizes  the  data  fi'om  this  procedure. 


Table  46: 

Predicting  Student  Perceptions  of  Competence  on  Command  and  Leadership:  Forcing  in  Environmental 
Variables  Not  In  Equation. _ 


Step 

Variable 

Multiple  R 

Simple  r 

R-squared 

R-squared 

Change 

4 

Teach  Methods 

.43 

.32 

.18 

.00 

5 

Technology 

.43 

.25 

.18 

.00 

6 

Research 

.43 

.25 

.18 

.00 

7 

Grading 

.44 

.24 

.19 

.01 

Teaching  Methods,  Technology,  and  Research  did  not  account  for  any  added  variance 
above  that  already  accounted  for  by  the  variables  already  in  the  equation.  The  Grading 
variable  did  not  enter  the  equation  either,  but  still  accounted  for  one  percent  (1  %)  above 
that  accoimted  for  after  the  other  variables  were  already  entered  into  the  equation.  This 
was  a  similar  situation  that  occurred  with  the  Air/Space  Power  variable  and  Research. 
Grading  has  a  larger  partial  regression  coefiScient  than  the  other  environmental  variables, 
except  Curriculum  and  Faculty.  This  means  that  after  statistical  adjustment  for  the  other 
independent  variables  in  the  equation.  Grading  affected  students’  perceptions  of 
competence  on  Command  and  Leadership  to  a  greater  degree  than  Teaching  Methods, 
Technology,  and  Research  affected  it.  Therefore,  Grading  can  still  account  for  a  small 
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portion  of  the  variance  even  after  the  other  variables  were  already  in  the  equation.  Notice 
that  the  Pearson  “r”s  for  the  variables  in  table  46  were  extremely  close,  suggesting  that 
these  variables  might  have  had  similar  predictive  powers. 

Table  47  compares  the  amount  of  variance  explained  by  each  environmental  variable 
when  they  were  entered  individually  after  inputs.  The  R-squared  values  in  the  second 
column  are  very  close  which  shows  that,  when  these  variables  were  entered  individually 
after  inputs  were  controlled,  the  predictive  power  of  each  was  similar.  These  figures 
corroborate  the  data  fi'om  tables  45  and  46,  revealing  that  out  of  the  sbc  environmental 
variables.  Curriculum  and  Faculty  had  the  most  predictive  power  once  inputs  were 
controlled.  However,  some  of  the  other  variables,  when  allowed  to  enter  the  equation 
first,  accounted  for  notable  amoimts  of  variance.  Grading,  for  example,  accounted  for 
seven  percent  (7  %)  of  the  variance  when  allowed  to  enter  the  equation  first,  which  was 
only  one  percent  (1  %)  lower  than  Faculty. 

Table  47: 


Predicting  Student  Perceptions  of  Competence  on  Command  and  Leadership: 
Enviromnental  Variables  Entered  Separately  After  Inputs. _ 


Variable 

Multiple  R 

R-squared 

R-squared 

Change 

Inputs  after 

.19 

.04 

.04 

Step  1 

Curriculum 

.38 

.14 

.10 

Teach  Methods 

.32 

.10 

.06 

Faculty 

.35 

.12 

.08 

Technology 

.29 

.08 

Grading 

.33 

.11 

hhhshhii 

Research 

.30 

.09 

.05 
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Predictions  on  Critical  Thinking 

The  results  of  the  regression  analysis  of  the  Critical  Thinking  variable  are  summarized 
in  table  48.  This  table  displays  four  input  variables  that  entered  the  equation  from  the  first 
block.  When  the  second  block  of  variables  were  entered,  two  environmental  variables 
emerged  from  the  analysis:  Curriculum  and  Teaching  Methods.  Two  of  the  input 
variables  that  entered  the  equation  in  the  first  block,  Age  and  Expectation  8,  were  the 
same  two  variables  that  appeared  in  earlier  analyses.  The  other  two  variables,  the  Mission 
Support  career  field  (information  management,  personnel,  finance,  etc.)  and  Expectation 
3,  “I  would  meet  other  people  and  learn  about  other  career  fields,”  require  some 
elaboration. 


Table  48: 


Predicting  Student  Perceptions  of  Competence  on  Critical  Thinking. 


Step 

Variable 

Multiple  R 

Simple  r 

R~squared 

R-squared 

Change 

Input 

1 

Exp  3 

.19 

.24 

.04 

.04 

2 

Support 

.24 

-.09 

.06 

.02 

3 

Age  2 

.27 

.12 

.07 

.01 

4 

Exp  8 

.30 

.17 

.09 

.02 

Environment 

5 

Curriculum 

.46 

.41 

.21 

.12 

6 

Teach  Methods 

.48 

.39 

.23 

.02 

Because  the  forward  selection  method  of  SPSS  enters  variables  in  the  equation  largely 
based  on  partial  regression  coefficients,  the  order  that  the  input  variables  entered  the 
equation  in  block  one  corresponded  to  those  coefficients.  The  partial  regression 
coefficients  for  the  input  variables  in  table  48  were  as  follows: 
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Expectation  3  3.111 840 

Support  -3.052972 

Age  2  2.219535 

Expectations  ,314900 

Notice  that  even  after  statistical  adjustment  for  the  other  variables  in  the  equation,  the 
Support  career  field  negatively  affected  students’  perceptions  of  competence  on  Critical 
Thinking.  Even  though  the  partial  regression  coefBcient  of  Support  was  negative,  it  was 
still  larger  than  either  Age  2  or  Expectation  8,  which  accounted  for  why  Support  entered 
the  equation  at  step  2.  This  means  that  respondents  who  expected  ACSC  to  provide  them 
with  a  chance  to  network,  who  were  not  in  the  Support  career  field,  who  were  36-39 
years  old,  and  who  expected  to  be  academically  challenged  at  ACSC  tended  to  score  their 
competence  on  Critical  Thinking  as  high. 

After  the  input  variables  were  accounted  for.  Curriculum  and  Teaching  Methods 
entered  the  equation  at  steps  5  and  6.  Input  variables  together  explained  nine  percent  (9 
%)  of  the  variance  in  the  equation.  Curriculum  explained  twelve  percent  (12  %)  of  the 
variance  above  that  explained  by  all  the  input  variables.  After  Curriculum  entered  the 
equation.  Teaching  Methods  only  added  two  percent  (2  %)  above  variance  explained  by 
inputs  and  Curriculum.  These  figures  indicate  that  Curriculum  had  the  most  predictive 
power  from  the  environmental  variable  block  after  inputs  entered  the  equation,  while 
Teaching  Methods  added  very  little.  Multicolinearity  allowed  Curriculum  to  account  for 
the  majority  of  the  variance,  leaving  little  left  for  Teaching  Methods. 

Table  49  shows  what  would  happen  if  the  environmental  variables  not  entered  in  the 
equation  were  forced  into  the  regression  after  inputs.  As  this  table  indicates.  Faculty 


148 


added  only  one  percent  (1  %)  above  Teaching  Methods,  while  Technology  and  Grading 
added  nothing  to  the  equation.  Research,  however,  added  one  percent  (1  %)  above 
Grading  when  forced  in  at  step  10.  Again,  this  resulted  because  Research  had  the  largest 
partial  regression  coefficient  (-.5245 1 7)  after  Faculty.  Notice  that  Research  had  a 
negative  effect  on  students’  perceptions  of  competence  on  Critical  Thinking  as  it  did  in  the 
Air/Space  Power  variable,  evidenced  by  the  negative  coefficient.  In  fact,  the  size  of  the 
regression  coefficient  indicates  a  much  stronger  negative  effect  on  Critical  Thinking  than 
was  found  in  Air/Space  Power,  where  the  regression  coefficient  was  only  -.143189. 


Table  49: 


Predicting  Student  Perceptions  of  Competence  on  Critical  Thinking:  Forcing  in  Environmental  Variables 
Not  In  Equation. _ 


Step 

Variable 

Multiple  R 

Simple  r 

R-squared 

R-squared 

Change 

7 

Faculty 

.49 

.31 

.24 

.01 

8 

Technology 

.49 

.23 

.24 

.00 

9 

Grading 

.49 

.22 

.24 

.00 

10 

Research 

.50 

.13 

.25 

.01 

In  order  to  see  how  much  variance  was  explained  by  each  individual  environmental 
variable  after  inputs  were  entered,  each  variable  was  entered  individually  into  a  regression 
equation  with  the  input  variables  controlled.  Table  50  shows  the  results  of  this  regression 
analysis.  The  input  variables  after  step  4  accounted  for  nine  percent  (9  %)  of  the  variance. 
The  R-squared  change  values  for  each  environmental  variable  reflect  the  difference 
between  each  environmental  variable  and  the  inputs  after  step  4.  This  provides  a  concise 
comparison  of  the  amount  of  added  variance  provided  by  each  environmental  variable 
after  inputs. 
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Table  50: 

Predicting  Student  Perceptions  of  Competence  on  Critical  Thinking:  Environmental 


Variables  Entered  Separately  After  Inputs. 


Variable 

Multiple  R 

R-squared 

R-squared 

Change 

Inputs  after 

.30 

.09 

.09 

Step  4 

Curriculum 

.46 

.21 

.12 

Teach  Methods 

.43 

.18 

.09 

Faculty 

.39 

.15 

.06 

Technology 

.36 

.13 

.04 

Grading 

.37 

.14 

.05 

Research 

.32 

.10 

.01 

Table  50  confirms  the  results  of  the  regression  analysis  fi’om  table  48,  with  the  highest 
R-squared  values  in  the  Curriculum  and  Teaching  Methods  variables.  Faculty, 
Technology,  and  Grading  added  similar  amounts  to  variance  explained  after  inputs  when 
they  were  entered  individually.  Finally,  Research  shows  a  small  R-squared  change  value 
of  one  percent  (1  %)  indicating  that,  although  Research  negatively  affected  the  outcome 
variable,  it  still  accounted  for  a  small  amount  of  variance  when  forced  to  enter  the 
equation  at  step  5,  after  inputs  were  controlled. 


In  conclusion,  a  comparison  of  the  regression  analyses  of  the  four  outcome  variables 
reveals  patterns  of  prediction.  For  all  four  outcome  variables,  the  input  variable  that 
always  entered  the  equation  in  the  first  block  was  Expectation  8,  “I  would  be  academically 
challenged  while  a  student  at  ACSC.”  This  seems  to  suggest  that  respondents  who  had 
greater  expectations  of  an  academically  challenging  program  were  more  likely  to  perceive 
themselves  as  having  higher  competence  on  all  four  outcome  variables.  Another  variable 
that  entered  the  equation  in  the  first  block,  except  in  Command  and  Leadership,  was  Age 
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2,  the  36-39  age  group.  According  to  these  data,  respondents  who  were  36-39  years  old 
were  more  likely  than  the  other  two  age  groups  to  score  themselves  highly  competent  on 
outcomes  except  for  Command  and  Leadership.  Two  career  fields  entered  the  regression 
equation  in  the  first  block  on  two  variables:  Medical/Legal  in  Air/Space  Power 
Applications  and  Support  (a  negative  effect)  in  Critical  Thinking.  These  data  reveal  that 
for  Air/Space  Power  Applications,  those  students  who  were  Medical/Legal  officers  were 
more  likely  to  perceive  themselves  as  highly  competent,  while  students  who  were  not 
Mission  Support  (information  management,  personnel,  finance)  were  more  likely  to  score 
themselves  highly  competent  on  the  Critical  Thinking  variable. 

After  controlling  for  input  variables,  the  variables  that  entered  the  equation  on  the 
second  block  also  show  similar  patterns  for  each  variable.  For  the  most  part.  Curriculum 
and  Teaching  Methods  entered  the  equation  with  Curriculum  accounting  for  most  of  the 
variance  before  Teaching  Methods  entered.  The  only  exception  to  this  pattern  was  with 
the  Command  and  Leadership  variable  where  Curriculum  and  Faculty  entered  the 
equation.  However,  Curriculum  still  accounted  for  the  majority  of  the  variance,  leaving 
very  little  left  to  Faculty.  The  Curriculum  variable  consisted  of  five  (5)  items  from  the 
questionnaire  (59-62,  67).  Items  59-62  dealt  with  assigned  books,  curriculum  flow,  the 
sytems  approach  to  problem  solving,  and  the  balance  between  academics  and  social 
fimctions.  Item  67  asked  students  the  extent  to  which  they  believed  ACSC  to  be  a 
graduate-level  program.  Since  the  Curriculum  variable  apparently  was  most  predictive  of 
outcome  variables,  it  was  necessary  to  run  regressions  between  individual  Curriculum 
items  and  the  four  outcome  variables.  The  results  of  these  analyses  revealed  that  item  61, 
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“The  systems  approach  to  problem  solving,”  accoimted  for  the  most  variance  in  all  four 
outcome  variables.  In  other  words,  out  of  the  Curriculum  items,  those  respondents  who 
rated  the  quality  of  the  systems  approach  to  problem  solving  as  high  were  most  likely  to 
score  themselves  as  highly  competent  in  outcome  variables.  After  item  61 ,  item  59, 
“assigned  books,”  entered  the  equation  except  in  Command  and  Leadership  where  item  61 
accounted  for  all  of  the  variance  (14  %).  The  assigned  books  item  added  a  little  variance 
above  the  systems  approach  item,  with  the  largest  percentage  (6  %)  of  added  variance  in 
the  Critical  Thinking  variable.  The  other  Curriculum  items  did  not  enter  the  equation, 
suggesting  they  did  not  add  anything  to  the  overall  variance.  From  these  data,  one  can 
infer  that,  out  of  all  of  the  items  on  the  questionnaire,  item  61  on  the  systems  approach  to 
problem  solving  was  most  predictive  of  students’  perceptions  of  competence  on  outcome 
items. 

The  research  question  that  covered  the  predictive  analysis  was:  “Holding  the  input 
variables  (demographics  and  student  expectations)  constant,  was  there  a  relationship 
between  environmental  variables  (teaching  methods  and  program  activities)  and 
outcomes?”  The  predictive  analysis  revealed  the  extent  to  which  environmental  variables 
affected  outcomes  after  input  variables  were  controlled.  By  controlling  for  the  potential 
bias  of  input  variables,  one  can  be  more  certain  that  the  effects  of  environmental  variables 
on  outcomes  were  true  effects.  However,  conclusions  of  the  predictive  analyses  must  be 
tempered  by  the  feet  that  there  may  be  some  important  student  input  characteristics  that 
were  not  controlled  in  the  analysis.  Even  though  pretest  data  are  known  to  have 
considerable  predictive  power  on  outcomes,  this  analysis  did  not  include  any  pretest 
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variables  in  the  inputs.  As  stated  earlier,  this  was  a  limitation  of  this  study.  What  can  be 
concluded  from  the  predictive  analysis  is  that,  after  controlling  for  the  effects  of  inputs, 
which  accounted  from  3  %  (Command  and  Leadership)  to  9  %  (Critical  Thinking)  of  the 
variance,  the  Curriculum  variable  added  from  10  %  (Command  and  Leadership)  to  15  % 
(Joint  Campaign)  above  inputs.  This  suggests  that  after  controlling  for  inputs, 
respondents  who  rated  Curriculum  (particularly  item  61)  as  high  were  more  likely  to 
perceive  their  competence  on  outcome  variables  as  high.  The  predictive  power  of  the 
Curriculum  variable  depended  on  which  outcome  was  the  dependent  variable.  Apparently, 
Curriculum  accoimted  for  the  most  variance  after  inputs  in  the  Joint  Campaign  variable, 
and  the  least  in  Command  and  Leadership.  This  is  probably  due  to  the  effects  of 
multicolinearity  where  variables  must  share  variance  with  the  other  variables  in  the 
equation.  Therefore,  the  amount  of  variance  accounted  for  by  Curriculum  depends  on  the 
other  variables  that  were  already  in  the  equation.  Finally,  the  results  of  the  analyses 
revealed  that  other  environmental  variables,  which  did  not  enter  the  equation  when  they 
must  share  variance  with  Curriculum,  would  account  for  some  of  the  variance  after  inputs 
if  they  were  allowed  to  enter  the  equation  at  the  first  step.  These  data  suggest  that, 
although  most  environmental  variables  did  not  enter  the  equation,  they  should  not  be 
discounted  and  their  predictive  power  should  be  examined  individually. 

Qualitative  Analysis 

Questions  89  and  90  on  the  questionnaire  were  open-ended  qualitative  questions. 

They  were  included  so  that  graduates  would  have  an  opportunity  to  address  issues  that 
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were  either  not  on  the  questionnaire  or  items  that  respondents  felt  needed  elaboration. 
This  brief  narrative  summarizes  the  qualitative  information  provided  by  respondents  in 
questions  89  and  90.  The  researcher  compiled  the  qualitative  data  into  a  single  document 
and  then  conducted  a  content  analysis  to  extract  patterns. 

Question  89  asked  students,  “In  one  or  two  sentences,  describe  how  you  would 
improve  Air  Command  and  Staff  College.”  Out  of  the  228  returned  questionnaires,  198 
respondents  (87  %)  wrote  comments  on  question  89.  Five  categories  of  comments 
emerged  as  a  result  of  the  content  analysis  (curriculum,  grading  and  distinguished 
graduates  program,  faculty,  research,  and  teaching  methods). 

Patterns  of  comments  on  the  curriculum  centered  around  three  main  topics:  teaching 
more  command  and  leadership  skills,  emphasizing  joint  campaign  planning  more,  and 
reducing  the  reading  workload  to  give  students  a  chance  for  more  analysis.  Respondents 
who  wrote  in  the  open-ended  portion  of  the  questionnaire,  believed  that  the  curriculum 
was  too  focused  on  air  campaign  planning  and  not  enough  on  command  and  leadership 
issues.  One  respondent  siunmed  it  up  like  this,  “As  a  commander  it  would  have  been 
appropriate  if  Air  Command  and  Staff  College  would  have  spent  some  curriculum  time 
teaching  something  besides  air  campaign  planning  which  I  suspect  less  than  10%  of  grads 
will  ever  do!”  In  addition  to  more  command  and  leadership,  respondents  also  believed 
that  the  school  needed  to  emphasis  joint  campaign  planning  more  because  many  of  them 
would  be  working  in  joint  assignments  after  graduation  fi’om  ACSC.  The  third  curricular 
pattern  revolved  around  the  reading  workload.  According  to  many  respondents,  the 
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amount  of  reading  was  unrealistic.  Graduates  believed  that  they  would  benefit  more  fi'om 
less  but  more  focused  reading,  giving  them  more  time  to  discuss  and  analyze  the  material. 

Comments  on  the  method  of  grading  at  ACSC  were  tied  to  the  distinguished  graduate 
(DG)  program  in  most  cases.  Of  the  50  respondents  who  made  comments  about  the 
grading  process,  30  of  them  (60  %)  recommended  that  the  DG  program  be  discontinued. 
Overall  comments  about  the  DG  program  revolved  around  standardization  of  grades. 
Respondents  believed  that  grading  was  subjective  and  too  dependent  on  the  course 
instructor.  Because  DG  was  determined  largely  by  grades,  many  graduates  perceived  the 
subjective  grading  method  to  be  unfair.  Respondents  also  believed  that  group  grades  on 
research  projects  were  an  unfair  practice  that  kept  many  individuals  (with  high  individual 
grades)  fi'om  becoming  distinguished  graduates.  While  these  comments  about  unfair 
grading  practices  and  the  DG  program  appear  to  be  from  those  who  did  not  achieve  DG 
status,  this  was  not  always  the  case.  Some  respondents  self-disclosed  that  they  were 
distinguished  graduates  fi'om  the  College  and  they  still  felt  that  the  grading  system  was 
“arbitrary.” 

Thirty-one  (31)  respondents  made  comments  about  the  faculty.  Of  those  thirty-one 
individuals,  twenty-six  (26)  wrote  that  the  faculty  was  unqualified  to  be  teaching  at  the 
College.  Comments  like,  “hire  instructors  with  more  experience,”  “bring  the  very  best 
officers  to  serve  as  iaculty,”  and  “hire  instructors  that  have  expertise  in  the  field  that  they 
are  teaching,”  were  typical  of  the  criticisms  about  feculty.  A  few  wrote  that  the  school 
should  hire  more  Ph.D.s,  but  most  were  concerned  that  ACSC  was  not  seeking  out  and 
hiring  the  most  qualified  feculty  members,  regardless  of  their  level  of  education. 
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The  research  component  of  the  College  was  another  topic  of  criticism.  Respondents 
(25)  who  commented  about  research  wrote  that  ACSC  research  methods  were  inadequate 
to  be  deemed  research.  The  frustration  seemed  to  stem  from  the  perceived  incongruence 
between  the  time  allotted  for  research  and  the  expectations.  Respondents  believed  that 
they  did  not  have  enough  time  to  pursue  their  research  or  interface  with  feculty  about  their 
projects.  In  addition,  many  commented  that  they  did  not  come  to  ACSC  with  the  skills 
necessary  to  do  research  but  training  was  not  available  to  them.  Yet,  the  College  placed  a 
great  deal  of  emphasis  on  research,  by  making  collaborative  projects  mandatory  and  by 
including  research  grades  in  criteria  for  DG.  The  disappointment  felt  by  many  respondents 
was  summed  up  by  one  graduate  who  said,  “They  want  a  high  quality  product  but  really 
keep  you  hxunping  with  large  reading  assignments.  This  leaves  too  little  time  for 
research.” 

The  final  topic  addressed  in  the  comments  to  question  89  was  teaching  methods.  Only 
a  small  number  of  respondents  (8)  wrote  about  teaching  methods  but  they  made  some 
valid  observations.  A  few  respondents  wrote  that  there  should  be  less  lectures  with  more 
seminar  work  at  the  College.  Some  of  those  who  made  comments  about  teaching 
methods  were  concerned  that  senior-level  Air  Force  leadership  did  not  come  to  ACSC  to 
speak.  Still  others  wrote  that  guest  lecturers  should  present  both  sides  of  their  topics  in 
order  to  provide  a  balance. 

While  question  89  was  designed  to  solicit  comments  about  how  ACSC  could  be 
improved,  the  open-ended  question,  #90,  was  intended  to  provide  respondents  with  an 
opportunity  to  elaborate  on  scaled  items  from  the  rest  of  the  questionnaire.  Question  90 
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states,  “Please  use  this  space  to  comment  on  any  of  the  items  on  this  questionnaire.  Be 
sure  to  specify  to  which  item  you  are  referring.”  The  content  analysis  placed  the 
comments  on  this  question  into  the  three  I-E-0  categories.  Questions  82-88  on  the 
questionnaire  corresponded  to  input  variables  of  demographics  and  student  expectations. 
In  the  open-ended  portion  of  the  questionnaire,  a  few  respondents  commented  that  they 
did  not  want  to  give  any  demographic  information  because,  “this  goes  beyond  statistics 
and  can  narrow  the  data  to  one  individual.”  Most  respondents  did  answer  the 
demographic  section  but  a  couple  refused  to  give  race  or  gender.  Those  who  chose  to 
comment  about  their  decision  not  to  supply  this  information  wrote  that  race  and  gender 
should  not  matter  in  a  survey.  In  addition  to  commenting  on  demographic  data,  some 
respondents  used  question  90  to  elaborate  about  the  student  expectation  portion  of  the 
questionnaire  (question  88).  Those  writing  about  this  area  (4  individuals)  gave  positive 
and  negative  responses.  Some  wrote  that  ACSC  was  not  what  they  expected,  but  they 
were  pleasantly  surprised  that  it  was  academically  challenging.  Conversely,  others  wrote 
that,  although  they  did  not  expect  to  have  a  lot  of  free  time  at  ACSC,  they  did  not  expect 
to  have  less  time  than  they  had  in  their  previous  jobs.  They  attributed  this  to  the  large 
number  of  assigned  reading  materials. 

The  comments  from  question  90  that  corresponded  to  the  environmental  variables 
(questions  50-81  on  the  questionnaire)  are  the  most  interesting.  Although  respondents 
elaborated  on  a  number  of  environmental  items,  four  items  stand  out.  The  largest  number 
of  individuals  (12)  chose  to  use  question  90  to  elaborate  on  item  73,  “To  what  extent 
should  grades  be  used  for  determining  distinguished  graduates?”  The  comments  on  this 
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item  mirror  those  already  described  from  question  89.  Graduates  who  wrote  about  this 
item  felt  that  grades  were  much  too  subjective  to  be  of  any  value  in  determining 
distinguished  graduates.  Some  respondents  recommended  that  the  program  be  eliminated. 
Another  item  upon  which  respondents  frequently  commented  was  item  74,  “To  what 
extent  have  you  found  the  ACSC  curriculum  CD-ROM  to  be  useful?”  Elaborations  of  this 
item  centered  around  whether  or  not  the  respondent  had  CD-ROM  access.  Most  said  that 
they  would  use  the  CD-ROM  if  they  had  access  either  at  home  or  work.  Two  other  items 
were  frequently  targeted  in  question  90:  item  75,  “What  did  you  think  about  the  amoimt 
of  assigned  reading?”  and  item  76,  “What  did  you  think  about  the  length  of  the  courses?” 
Again,  written  comments  about  these  topics  mirrored  those  in  question  89.  Respondents 
felt  that  there  was  too  much  reading  (item  75),  and  that  some  courses  were  too  long, 
while  others  were  too  short  (item  76). 

Respondents  commenting  about  outcome  variables  (questions  5-49)  were  mostly 
concerned  about  the  survey’s  methodology.  Twenty-seven  (27)  individuals  wrote  that 
they  believed  that  the  survey  was  biased  because  it  did  not  ask  students  for  their 
perceptions  of  their  competence  prior  to  attending  ACSC.  Comments  such  as, 
“competence  was  not  necessarily  gained  at  ACSC,”  and  “  many  questions  I  can  rate  as  5, 
but  not  because  of  ACSC,”  were  typical  of  the  remarks  made  about  items  5-49  on  the 
questionnaire.  While  these  observations  are  valid,  the  survey’s  methodology  and 
limitations  could  not  be  fully  explained  within  the  scope  of  the  questionnaire.  Although 
there  were  many  people  who  seemed  to  be  concerned  about  the  survey’s  validity,  these 
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individuals  took  the  time  to  answer  all  the  questions  and  to  elaborate  freely  in  the  open- 
ended  portion  of  the  questionnaire. 

In  conclusion,  open-ended  comments  on  the  questionnaire  supplemented  the 
quantitative  analysis  by  addressing  many  of  the  same  issues.  The  relational  analysis 
revealed  that  Command  and  Leadership,  an  outcome  variable  was,  more  often  than  not, 
the  least  associated  with  input  (demographic  and  student  expectations)  variables.  Also, 
when  comparing  correlations  between  environmental  and  outcome  variables.  Command 
and  Leadership  was  one  of  the  least  associated  with  the  environmental  variables. 

Similarly,  a  large  number  of  respondents  wrote  that  not  enough  Command  and  Leadership 
was  taught  at  ACSC.  This  was  one  of  the  most  criticized  elements  that  emerged  from  the 
content  analysis  on  the  qualitative  data.  Other  patterns  from  the  qualitative  analysis  also 
mirror  those  from  the  relational  analysis.  For  example.  Research  (an  environmental 
variable)  was  a  major  topic  in  students’  responses  on  the  qualitative  portion  of  the 
questionnaire.  Correlations  between  Research  and  the  outcome  variables  revealed  that 
Research  had  the  least  association  with  outcomes.  There  are  many  more  examples  of  how 
the  qualitative  data  elaborates  on  the  descriptive  and  relational  analyses.  Therefore,  the 
qualitative  data  add  greatly  to  the  existing  quantitative  information  which  has  already  been 
discussed.  It  provides  another  element  of  understanding  about  ACSC  and  how  the  entire 
program  can  be  improved.  Portions  of  this  qualitative  information  will  be  used  along  with 
the  quantitative  data  in  the  final  conclusions  and  recommendations  in  chapter  five. 
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CHAPTER  FIVE 

CONCLUSIONS  AND  RECOMMENDATIONS 

This  chapter  covers  conclusions  and  recommendations  suggested  by  the  results  and 
findings  outlined  in  chapter  four.  In  order  to  present  conclusions  and  recommendations,  it 
is  necessary  to  provide  a  brief  summary  of  the  findings.  After  the  summary,  a  discussion 
will  follow  that  will  tie  the  conclusions  suggested  by  the  findings  back  to  the  problem  of 
the  study.  Once  conclusions  are  outlined,  the  discussion  will  turn  to  the  recommendations 
and  implications  for  Air  Command  and  Staff  College.  Implications  for  future  research  and 
policy  will  be  outlined.  Finally,  limitations  of  the  study  will  be  addressed. 

Summary  of  the  Major  Findings 

This  study  was  interested  in  determining  the  overall  effectiveness  of  Air  Command  and 
Staff  College  based  on  the  perceptions  of  recent  Air  Force  graduates  of  the  College 
(n=395).  Effectiveness  was  defined  as  the  extent  to  which  students  rated  their 
competencies  on  outcome  variables  and  how  students’  perceptions  of  the  quality  of 
teaching  methods  and  program  activities  affected  their  perceived  competency  on 
outcomes.  Based  on  the  data  fi-om  a  90-item  questionnaire  titled,  “Student  Perceptions  of 
Program  Effectiveness  Questionnaire,”  the  researcher  analyzed  student  perceptions  of 
three  types  of  variables— inputs,  environment,  and  outcomes.  Input  variables  consisted  of 
demographic  fectors  and  student  expectations.  Environmental  variables  were  the  teaching 
methods  and  program  activities  to  which  students  were  exposed  while  at  ACSC. 
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Outcomes  were  defined  as  the  course  objectives  derived  by  distilling  information  from 
course  syllabi.  Various  statistical  methods  were  used  to  analyze  the  data,  starting  with 
descriptive  information  in  the  form  of  means,  standard  deviations,  and  percentages.  Next, 
the  relationships  between  input  variables  and  outcomes  were  reported  using  the  cross¬ 
tabulation  method.  Cross-tabulations  were  used  to  analyze  the  percentages  of  respondents 
who  scored  themselves  highly  competent  (an  average  of  4  and  5)  on  outcome  variables  by 
all  of  the  demographic  fectors  (age,  gender,  race,  career  field,  Master  Degree 
concentration,  and  commission  source)  and  the  nine  student  expectations.  The 
relationships  between  environmental  variables  and  outcomes  were  analyzed  with  Pearson 
“r”s.  Significant  relationships  at  the  .01  level  of  significance  were  reported.  Finally, 
multiple  regression  analysis  was  used  to  determine  the  predictive  power  of  environmental 
variables,  after  controlling  for  the  effects  of  potentially  biasing  input  variables.  The  results 
of  the  data  analysis  on  each  outcome  variable  were  reported.  As  can  be  seen  from  this 
brief  synopsis  of  statistical  methods  used,  this  study  was  designed  to  provide  breadth 
rather  than  depth  by  analyzing  the  data  with  different  and  increasingly  more  sophisticated 
methods. 

Descriptive  Findings.  The  first  question  posed  by  this  dissertation  focused  on 
describing  the  demographic  factors  and  student  expectations  of  respondents.  Descriptive 
demographic  data  revealed  that  graduates  from  the  Academic  Year  (AY)  1994-1995  class 
at  ACSC  tended  to  be  white  males  in  their  middle  thirties  (36-39).  Respondents  were 
more  likely  to  be  in  a  non-operational  career  field,  most  often  in  a  Mission  Support  job 
(information  management,  personnel,  finance,  etc.)  with  an  earned  Master  Degree,  more 
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often  than  not,  in  a  Social  Sciences  field.  Officers  fi’om  this  class  tended  to  have  been 
commissioned  through  the  Reserve  Officer  Training  Corp  (ROTC),  with  Air  Force 
Academy  graduates  making  up  the  smallest  percentage  (19.9%)  of  ACSC  graduates  fi'om 
this  class.  An  analysis  of  student  expectations  revealed  that  officers  generally  expected 
ACSC  to  be  academically  challenging  and  to  provide  them  with  a  chance  to  “network” 
with  their  fellow  officers.  Most  officers  also  expected  ACSC  to  be  important  for  their 
career  progression. 

The  second  research  question  asked  how  students  rated  the  teaching  methods  and 
program  activities  (environmental  variables)  at  ACSC.  The  following  table  summarizes 
the  findings  by  showing  overall  means  of  each  environmental  variable.  Means  shown 
represent  collapsed  values  derived  by  averaging  the  questionnaire  items  indicated.  The 
scale  consisted  of  five  levels:  (1)  “unsatisfactory,”  (2)  “marginal,”  (3)  “satisfectory,”  (4) 
“excellent,”  (5)  “outstanding.”  For  a  more  detailed  examination  of  the  data,  please  refer 
to  the  descriptive  analysis  section  of  chapter  four. 


Table  51: 


Variable 

Overall  Means 

Teaching  Methods 
(items  50-53) 

3.32 

Technology 
(items  54-58;  74) 

3.05 

Curriculum 
(items  59-62;  67) 

3.47 

Research 
(items  63,  68,  69) 

2.55 

Faculty 

(items  64,  65,  70) 

3.33 

Grading 

(items  66,  71-73) 

2.91 
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This  table  shows  that  students  perceived  the  overall  quality  of  Curriculum  items  to  be 
the  highest,  relative  to  the  other  items,  followed  by  Teaching  Methods  and  Faculty. 
Research  and  Grading  items  were  perceived  as  being  the  lowest  rated  items  overall.  To 
elaborate  on  these  data,  respondents  generally  rated  guest  lecturers  as  being  of  higher 
quality  than  instructors.  They  did  not  like  the  computerized  readings,  the  amount  of 
reading  assigned,  group  research,  or  grading  procedures.  Students  wanted  to  see  more 
Command  and  Leadership  content  incorporated  into  the  curriculum  and  more  time, 
training  and  support  for  their  research  projects.  Civilian  faculty  members  were  rated  as 
being  of  higher  quality  than  military  faculty  members,  but  most  students  said  that  the 
balance  between  civilian  and  military  faculty  was  “about  right.”  In  their  written 
comments,  students  severely  criticized  the  way  in  which  grades  were  used  to  determine 
distinguished  graduates,  which  may  have  led  them  to  say  they  believed  that  pass/fail  grades 
were  more  appropriate  at  ACSC  than  letter  grades. 

The  third  question  posed  by  this  study,  and  the  last  descriptive  question,  asked  how 
students  rated  their  competency  on  outcome  variables  at  the  point  in  time  when  they 
completed  ACSC.  Again,  the  means  represent  averaged  means  on  the  items  indicated. 

The  scale  consisted  of  five  levels;  (1)  “None,”  (2)  “Very  little,”  (3)  “Some,”  (4) 
“Considerable,”  (5)  “Great.”  According  to  the  descriptive  data  on  outcome  variables 
presented  in  table  52,  students  rated  their  competency  close  to  “considerable,”  on 
Air/Space  Power  and  Critical  Thinking.  The  lowest  perceived  competence  was  in  the 
Command  and  Leadership  variable,  which  fell  almost  halfway  between  “some 
competence”  and  “considerable  competence.”  Students  perceived  themselves  as  having  a 
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slightly  higher  competence  in  Joint  Campaign  than  in  Command  and  Leadership.  Again, 
for  specific  information  on  individual  items,  please  refer  to  tables  25-28  in  the  descriptive 
section  of  chapter  four. 


Table  52: 


Overall  Means  Comnarec 

i  on  Outcome  Variables 

Variable 

Overall  Means 

Joint  Campaign 
(items  35-45) 

3.78 

Air/Space  Power 
(items  46-49) 

4.04 

Command  &  Lead 
(items  5-16) 

3.69 

Critical  Thinking 
(items  17-34) 

3.90 

Relational  Findings.  The  first  relational  question  (question  four)  was  concerned  with 
the  relationship  between  input  variables  (demographics  and  student  expectations)  and 
outcomes.  The  data  were  analyzed  using  cross  tabulations  and  reported  in  percentages. 
The  analysis  revealed  that  across  all  demographic  categories  and  student  expectations,  the 
largest  percentage  of  respondents  scoring  themselves  as  highly  competent  on  outcome 
variables  was  in  the  Air/Space  Power  variable.  Conversely,  the  smallest  percentage  was  in 
the  Command  and  Leadership  variable.  In  all  outcome  variables,  except  in  Critical 
Thinking,  women  tended  to  perceive  themselves  as  more  competent  on  all  of  the  outcome 
variables  than  did  males.  Afiican- American  officers  tended  to  score  themselves  higher  on 
Joint  Campaign  and  Air/Space  Power  than  did  other  race  categories,  but  Hispanic  and 
Native- American  officers  scored  themselves  more  competent  in  Command  and  Leadership 
and  Critical  Thinking  than  did  other  race  categories.  Operations  officers  (pilots. 
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navigators,  missileers,  etc.),  in  all  categories  except  Air/Space  Power,  scored  themselves 
less  competent  than  those  in  other  career  fields.  Those  in  the  Medical/Legal  career  field, 
however,  tended  to  perceive  themselves  as  more  competent  on  all  four  outcome  variables 
than  did  those  in  other  career  fields.  Similarly,  the  data  analysis  on  the  percentage  of 
respondents  scoring  high  on  outcome  variables  by  Master  Degree  Concentration  revealed 
that  officers  with  Medical/Legal  degrees  tended  to  perceive  themselves  as  more  competent 
on  all  four  outcomes  than  other  degree  categories.  Also,  obtaining  a  Ph.D.  did  not  appear 
to  increase  students’  perceptions  of  competence  on  outcomes.  Comparing  percentages  of 
respondents  scoring  high  on  outcome  variables  by  Commission  Source  and  Student 
Expectations  revealed  that  students’  perception  of  their  competence  on  outcomes  did  not 
vary  considerably  across  commission  sources  or  expectations.  These  data  seem  to  indicate 
that  respondents’  perceived  level  of  competencies  on  outcome  variables  were  not  very 
affected  by  their  source  of  commission  or  their  expectations. 

Question  five  on  this  dissertation  was  concerned  with  the  relationship  between 
environmental  variables  (teaching  methods  and  program  activities)  and  student  outcomes. 
The  data  were  analyzed  using  Pearson  “r”s.  The  results  of  the  data  analysis  on  the 
relationships  between  environmental  and  outcome  variables  revealed  that  environmental 
variables  were  associated  with  outcomes  (significant  at  the  .01  level),  fi’om  most  to  least, 
in  the  following  manner:  Curriculum,  Teaching  Methods,  Faculty,  Technology,  Grading, 
and  Research.  The  largest  correlation  was  between  Curriculum  and  the  Joint  Campaign 
variable  (r=.46),  and  the  smallest  between  Research  and  Critical  Thinking  (r=.13).  There 
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was  no  relationship  between  Research  and  the  Air/Space  Power  variable  (r=.10  which  was 
not  significant  at  the  .01  level). 

Predictive  Findings.  Question  sk  posed  in  chapters  one  and  three  frames  the 
predictive  question  which  asks  what  were  the  effects  of  environmental  variables  on 
outcomes  after  controlling  for  the  effects  of  input  variables  (demographics  and  student 
expectations).  Through  multiple  regression  analysis  and  the  Statistical  Package  for  the 
Social  Sciences  (SPSS),  the  researcher  entered  variables  in  two  blocks,  first  analyzing  the 
overall  predictability  of  input  variables.  Once  input  variables  were  accounted  for,  the 
second  block  of  variables  consisting  of  environmental  variables  was  entered.  This  method 
aUowed  the  researcher  to  determine  the  true  effects  of  environmental  variables  on 
outcomes  by  first  controlling  for  the  potentially  biasing  effects  of  input  variables.  The  R- 
squared  change  values  in  table  53  indicate  to  what  extent  variation  in  outcomes  was 
explained  by  the  variables  and  in  what  order  they  entered  the  equation.  For  example,  for 
the  Joint  Campaign  variable.  Age  2  entered  the  equation  first,  accoimting  for  three  percent 
(3  %)  of  the  variation,  followed  by  Expectation  8,  which  added  five  percent  (5  %)  of  the 
explained  variance  above  what  could  be  explained  by  Age  2.  Once  these  input  variables 
were  in  the  equation.  Curriculum  and  Teaching  Methods  entered.  For  the  Joint  Campaign 
variable.  Curriculum  accounted  for  fifteen  percent  (15%)  of  the  variance  over  what  could 
be  attributed  to  input  variables.  Teaching  Methods,  in  this  instance,  only  added  three 
percent  (3%)  over  Curriculum.  In  other  words.  Curriculum  accoimted  for  most  of  the 
explained  variance  above  inputs,  leaving  Teaching  Methods  with  only  a  small  portion  of 


the  variance. 
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Table  53: 


Summary  of  Amounts  of  Variation  Explained  bv  Inputs  and  Environmental 
Variables.  _ _ _ 


Outcome  Variable 

R-squared  Change 
Inputs 

R-squared  Change 
Environment 

Joint  Campaign 
(items  35-45) 

Age  2  =  .03 

Exp  8  =  .05 

Curriculum  =  .15 
Teaching  Methods  =.03 

Air/Space  Power 
(items  46-49) 

Med/Legal  =  .03 

Age  2  =  .03 

Exp  8  =.02 

Curriculum  =  .13 
Teaching  Methods=.02 

Command  &  Lead 
(items  5-16) 

Exp  8  =  .03 

Curriculum  =  .10 
Faculty  =  .04 

Critical  Thinking 
(items  17-34) 

Exp  3  =  .04 
Support  =  .02 

Age  2  =  .01 

Exp  8  =  .02 

Curriculum  =  .12 
Teaching  Methods=.02 

Table  53  shows  that  of  all  of  the  input  variables,  Expectation  8,  “I  would  be 
academically  challenged  while  a  student  at  ACSC,”  had  some  predictive  power  for  all  four 
outcome  variables,  ranging  from  two  percent  (2%)  in  Air/Space  Power  and  Critical 
Thinking  to  five  percent  (5%)  in  Joint  Campaign.  Notice  that  Expectation  8,  although 
occurring  in  all  four  variables,  entered  the  equation  only  after  other  input  variables,  except 
in  Command  and  Leadership  where  it  was  the  only  input  variable  to  have  any  predictive 
power.  Age  2,  the  36-39  age  group,  also  occurred  in  all  outcomes  except  Command  and 
Leadership.  Age  2  added  from  one  percent  (1  %)  of  the  variance  (Critical  Thinking)  to 
three  percent  (3%)  in  Joint  Campaign  and  Air/Space  Power.  In  other  words,  students 
who  expected  to  be  academically  challenged  and  who  were  in  the  36-39  age  group  were 
more  likely  to  score  themselves  as  highly  competent  in  most  outcomes,  except  Command 
and  Leadership.  Two  career  fields  emerged  from  the  data  analysis  as  having  predictive 
power  but  only  for  two  outcome  variables.  Officers  who  were  in  the  Medical/Legal  career 
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field  were  more  likely  to  score  themselves  highly  competent  on  Air/Space  Power 
Applications,  while  those  graduates  who  were  not  in  the  Support  career  field  were  more 
likely  to  rate  themselves  high  on  competence  in  Critical  Thinking  (an  inverse  relationship). 

After  controlling  for  input  variables,  the  regression  equations  showed  that  Curriculum 
accoimted  for  the  most  variance  in  all  four  outcome  variables,  ranging  IBrom  ten  percent 
(10  %)  in  Command  and  Leadership  to  fifteen  percent  (15  %)  in  Joint  Campaign. 
Following  Curriculum,  Teaching  Methods  added  a  small  amount  in  every  equation  from 
two  percent  (2  %)  in  Air/Space  Power  and  Critical  Thinking  to  three  percent  (3  %)  in 
Joint  Campaign.  In  the  Command  and  Leadership  variable,  Faculty  replaced  Teaching 
Methods  where  it  accounted  for  four  percent  (4  %)  of  the  variance  over  that  which  could 
be  explained  by  Curriculum.  These  data  indicate  that  after  controlling  for  input  variables, 
respondents  who  rated  the  quality  of  Curriculum  as  high  tended  to  score  themselves  as 
highly  competent  in  the  four  outcome  variables.  In  other  words,  of  the  she  environmental 
variables  examined  in  this  study,  Curriculum  emerged  as  explaining  the  most  variance  in  all 
four  outcome  variables  after  the  potentially  biasing  effect  of  input  variables  were 
controlled. 

This  summary  highlighted  the  major  findings  of  the  data  analyses.  It  was  not  intended 
to  repeat  all  of  the  results  from  chapter  four,  but  was  meant  to  direct  the  reader’s  attention 
back  to  the  research  questions  and  the  overarching  themes  that  emerged  from  the  data. 

The  sections  that  follow  will  illuminate  some  possible  explanations  for  these  findings  and 
suggest  recommendations  and  implications  for  policy  changes  at  ACSC. 
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Conclusions 

The  mission  of  Air  Command  and  Staif  College  is  “to  educate  mid-career  ofiBcers  to 
develop,  advance,  and  apply  air  and  space  power  in  peace  and  war.”  While  this  mission 
statement  outlines  the  major  focus  of  the  College,  this  study  was  also  concerned  with 
outcomes  that  were  not  explicitly  stated.  The  four  outcome  variables  not  only  included 
Air/Space  Power  Applications,  but  also  looked  at  the  Joint  Campaign  Planning  Process 
and  developmental  goals  such  as  Critical  Thinking  and  Command  and  Leadership.  The 
researcher  determined  through  careful  analyses  of  course  syllabi  and  collaboration  with 
key  leaders  at  ACSC  that  these  four  outcomes  were  central  to  ACSC’s  program.  By 
looking  at  the  data  presented  in  this  study,  one  can  readily  see  that,  although  students 
scored  their  competency  on  Air/Space  Power  as  the  highest  among  the  outcomes,  they 
perceived  themselves  as  being  highly  competent  in  all  four  outcomes.  This  is  not 
surprising  given  the  feet  that  ACSC  students  tended  to  be  mature,  educated,  and 
motivated  individuals  who  have  been  highly  successful  in  their  respective  career  fields. 
Therefore,  one  explanation  for  the  high  level  of  perceived  competence  may  be  because 
those  chosen  to  attend  the  College  were  high  achievers  in  the  first  place.  Another  possible 
explanation  is  that  ACSC  was  having  an  effect  on  those  who  graduate  fi’om  the  program. 
This  study  was  designed  to  minimize  the  effect  of  the  things  students  brought  with  them  to 
the  College  by  controlling  for  input  variables  (demographics  and  student  expectations)  so 
that  any  conclusions  made  about  the  program’s  effect  on  outcomes  could  be  claimed  as 
true  effects.  The  conclusions  presented  in  this  section  will  highlight  the  major  findings  of 
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the  study  and  offer  elaborations  about  how  ACSC  affected  students  as  a  result  of  their 
participation  in  this  program. 

The  Effect  of  Inputs  on  Outcomes.  The  demographic  characteristics  of  the 
respondents  to  this  study  were  already  revealed,  but  what  was  the  profile  of  students  who 
perceived  themselves  as  highly  competent  across  most  of  the  four  outcome  variables? 
Based  on  correlational  data  between  demographics  and  outcome  variables,  students  who 
perceived  high  competence  on  outcomes  tended  to  be  between  the  ages  of  36-39,  be  an 
Afiican- American  female  in  the  Medical/Legal  career  field  (having  earned  a  Medical/Legal 
degree).  This  individual  tended  to  have  an  education  level  above  the  Master  level  (Master 
Plus)  and  be  a  ROTC  graduate.  This  profile  was  presented  to  show  the  dichotomy 
between  the  respondents  and  those  students  who  perceived  themselves  as  being  highly 
competent  on  outcome  variables.  These  data  seem  to  suggest  that  if  students  like  this  one 
could  be  found  in  the  population,  they  would  do  well  at  ACSC.  Of  course,  a  student  such 
as  this  one  does  not  exist  in  the  study  group  or  in  the  Air  Force.  Yet,  one  can  infer  from 
this  comparison  that  the  College  may  be  meeting  the  needs  of  its  minority  students, 
especially  black  females.  This  conclusion  must  be  tempered  by  the  feet  that  this  group 
represents  a  small  portion  of  respondents  and  that  individuals  who  choose  the  Air  Force  as 
a  career  may  not  be  typical  of  those  found  in  the  population  as  a  whole.  The  data  also 
seem  to  suggest  that  those  in  the  Medical/Legal  career  field  would  do  well  at  ACSC. 
Again,  these  individuals  represent  only  a  small  portion  (n=7)  of  respondents.  In  addition, 
those  in  the  Medical  and  Legal  professions  have  specialized  training  which  might  give 
them  an  edge  over  others  when  attending  a  program  such  as  ACSC.  Another  plausible 
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explanation  may  be  that  ACSC  is  serving  well  the  Medical  and  Legal  officers  who  attend 
the  program.  However,  if  the  goal  of  ACSC  is  to  address  the  needs  of  all  of  their 
students,  it  is  important  for  decision-makers  to  know  what  student  characteristics  were 
possessed  by  those  who  perceived  high  competence  on  outcome  variables. 

A  more  in-depth  synthesis  of  correlational  data  between  inputs  and  outcomes  revealed 
several  notable  conclusions.  While  those  in  the  36-39  age  group  perceived  themselves 
highly  competent  on  the  majority  of  outcomes,  they  perceived  the  least  competence  in 
Command  and  Leadership.  Students  in  the  40  and  over  age  group,  however,  perceived 
themselves  highly  competent  in  Command  and  Leadership,  but  the  least  competent  in 
Critical  Thinking  and  Joint  Campaign.  A  possible  explanation  for  this  is  that  older  officers 
have  more  confidence  in  their  abilities  as  leaders  than  other  age  groups  either  because  of 
their  life  experiences  or  work-related  knowledge.  Those  students  who  were  35  and  under 
perceived  the  least  competence  in  Air/Space  Power  and  in  Command  and  Leadership. 

Just  as  older  officers  perceived  themselves  highly  competent  in  Command  and  Leadership, 
it  appears  as  if  yoimger  officers  did  not  feel  as  competent  in  this  area  as  other  age 
categories.  This  may  be  because  yoimger  officers  do  not  have  as  much  life  experiences  as 
their  older  contemporaries.  Similarly,  younger  officers  may  not  feel  as  competent  in 
Air/Space  Power  as  other  age  groups  because  they  may  not  have  as  much  total  life 
experiences  outside  their  professional  development  as  older  officers.  One  might  conclude 
from  these  data  that  the  various  age  groups  perceived  their  competencies  differently 
depending  on  which  outcome  is  being  investigated.  While  the  school  is  addressing  the 
needs  of  the  mainstream  (36-39  age  group),  they  may  not  be  meeting  the  needs  of  outlier 
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groups  in  Command  and  Leadership  (35  and  under)  and  Critical  Thinking  (40  and  over). 
These  data  seem  to  indicate  that  younger  officers  need  Command  and  Leadership  training, 
while  older  officers  lack  critical  thinking  skills.  Knowing  this  information  provides  ACSC 
with  an  opportunity  to  address  those  needs. 

Interesting  findings  also  emerged  from  the  data  analysis  on  gender  and  race.  Females 
and  African-Americans  emerged  as  the  top  groups  scoring  themselves  highly  competent  in 
the  majority  of  outcomes.  This  might  be  explained  by  concluding  that  female  and  African- 
American  officers  arrived  at  the  school  with  higher  perceptions  of  their  abilities.  It  is 
possible  that  the  Air  Force  only  attracts  members  of  these  two  minority  groups  who 
already  perceive  themselves  as  highly  competent.  It  is  also  possible  that  women  and 
African-American  officers  are  better  learners  who  are  motivated  to  do  well  at  the  College. 
One  might  also  conclude  that  ACSC  is  doing  a  good  job  of  meeting  the  needs  of 
minorities,  especially  women  and  African-American  officers.  However,  other  data  do 
indicate  more  stereotypical  findings.  For  example,  even  though  female  respondents,  on 
the  average,  scored  their  competencies  as  high  on  almost  all  outcome  variables,  they  did 
not  perceive  themselves  as  having  high  competence  in  Critical  Thinking.  This  seems  to 
suggest  that  ACSC  needs  to  look  at  how  to  instill  in  female  officers  a  higher  perception  of 
competence  in  critical  thinking  skills. 

Another  finding  based  on  correlational  data  between  demographics  and  outcome 
variables  involved  the  career  field  and  the  Master  Degree  concentration  in  which 
perceptions  of  high  competence  were  highest.  Students  who  perceived  their  competence 
as  high  in  all  four  outcomes  were  in  the  Medical/Legal  career  field  and  degree 
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concentration.  This  seems  to  suggest  that  officers  in  the  Medical/Legal  career  field 
performed  better  at  ACSC  than  those  in  other  career  fields  at  ACSC.  This  observation 
must  be  moderated  by  two  things:  the  feet  that  the  Medical/Legal  sub-population  was 
small  (n=7)  and  that  given  the  nature  of  Medical  and  Legal  training,  it  is  possible  that 
these  officers  were  better  learners.  One  may  also  conclude  that  ACSC  is  meeting  the 
needs  of  the  Medical  and  Legal  officers  which  allows  them  to  perform  well  while  a  student 
in  the  program.  What  was  more  notable  fi'om  the  correlational  data  between  career  fields 
and  outcomes  was  the  perceived  competence  level  of  Operations  officers.  Excluding  the 
Air/Space  Power  variable,  officers  in  the  Operations  career  field  (pilots,  navigators, 
missileers,  etc.)  perceived  their  competence  in  outcome  variables  as  lowest  among  the 
career  fields  presented.  Even  though  Operators  did  not  make  up  the  majority  of  students 
at  the  College,  they  are  a  central  element  of  the  Air  Force.  It  makes  sense  that  they  should 
score  themselves  highly  competent  in  Air/Space  Power  because  of  the  nature  of  their  jobs 
which  deals  with  this  area  on  a  daily  basis.  Perhaps  the  fact  that  Operators  scored 
themselves  least  competent  in  all  outcomes  except  Air/Space  Applications  suggests  to 
ACSC  that  the  focus  of  their  program,  at  least  for  this  group,  should  move  in  the  direction 
of  Command  and  Leadership,  Joint  Campaign,  and  Critical  Thinking. 

The  data  on  the  correlation  between  demographics  and  Commission  Source  revealed 
that  those  who  were  commissioned  through  ROTC  were  most  likely  to  score  themselves 
high  on  all  outcomes  except  Critical  Thinking.  Air  Force  Academy  graduates  (USAFA) 
tended  to  score  themselves  highest  on  the  Critical  Thinking  outcome  variable.  These  data 
seem  to  suggest  that  those  who  were  commissioned  through  ROTC  may  need  to  improve 
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their  critical  thinking  skills  and  that  ACSC  might  be  able  to  provide  the  necessary  training. 
Since  those  commissioned  through  ROTC  make  up  the  majority  of  respondents  (and  the 
study  group  as  weU),  it  behooves  the  College  to  investigate  this  fiirther. 

When  one  reviews  the  Expectations  that  were  most  associated  vvdth  students’ 
perceptions  on  outcome  variables,  it  differs  across  outcomes.  For  example,  those  who 
scored  themselves  high  on  the  Joint  Campaign  variable  were  more  likely  to  choose 
Expectation  1,  “ACSC  would  improve  my  chances  for  ftiture  promotions,”  and 
Expectation  6,  “I  would  spend  a  great  deal  of  time  socializing.”  Conversely,  students  who 
scored  themselves  high  on  the  Air/Space  Power  variable  tended  to  choose  Expectation  9, 
“Other  (please  specify).”  Those  scoring  themselves  highly  competent  on  Command  and 
Leadership  tended  to  choose  Expectation  8,  “I  would  be  academically  challenged  while  a 
student  at  ACSC.”  and  those  with  perceptions  of  high  competence  on  Critical  Thinking 
chose  Ejqjectation  7,  “I  would  learn  very  little  while  a  student  at  ACSC.”  As  one  can  tell, 
students’  choice  of  expectations  varied  depending  on  outcome.  Yet,  this  is  valuable 
information  because  expectations  provide  additional  insight  into  how  inputs  affect 
outcomes.  For  example,  if  one  were  to  determine  that  Command  and  Leadership  needed 
to  be  en:5)hasized,  one  might  conclude  from  the  data  that  by  finding  a  way  to  instill  in 
students  an  expectation  of  being  academically  challenged,  students’  perceptions  of  their 
competence  on  Command  and  Leadership  would  subsequently  improve. 

The  first  step  in  the  regression  analyses  on  all  four  outcomes  was  to  enter  the  input 
variables  as  a  block.  The  results  of  this  procedure  yielded  iiformation  about  which  input 
variables  were  most  predictive  of  outcomes.  As  stated  in  the  summary  and  presented  in 
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table  53,  of  all  of  the  input  variables,  Expectation  8,  “I  would  be  academically  challenged 
while  a  student  at  ACSC,”  had  some  predictive  power  for  all  four  outcome  variables.  It 
accounted  for  the  most  variance  in  Joint  Campaign  (5  %)  and  the  least  in  both  Command 
and  Leadership  and  Critical  Thinking  (2  %).  However,  Expectation  8  entered  the 
equation  first  in  only  one  variable,  Command  and  Leadership.  As  one  can  tell  by  looking 
at  the  data  presented  in  table  53,  the  regression  analyses  on  input  variables  yielded 
different  results  for  each  outcome  variable.  For  the  Joint  Campaign  variable.  Age  2  (36- 
39  age  group)  entered  the  equation  first,  accounting  for  three  percent  (3  %)  of  the 
variance,  followed  by  Expectation  8,  which  accounted  for  five  percent  (5  %)  of  the 
explained  variance  over  what  could  be  attributed  to  Age  2.  This  means  that  those  who  fell 
within  the  36-39  age  group  and  expected  to  be  academically  challenged  while  a  student  at 
ACSC  were  more  likely  to  score  themselves  highly  competent  in  Joint  Campaign.  For  the 
Air/Space  Power  Applications  outcome  variable,  three  input  variables  were  predictive  of 
perceived  high  competence:  the  Medical/Legal  career  field,  Age  2,  and  Expectation  8.  In 
other  words,  those  who  were  in  the  Medical/Legal  career  field,  in  the  36-39  age  group, 
and  who  expected  to  be  academically  challenged  while  a  student  at  ACSC  were  more 
likely  to  score  themselves  highly  competent  on  Air/Space  Power.  Conversely,  only  one 
input  variable  appeared  to  be  predictive  of  high  perceived  competence  on  Command  and 
Leadership-Expectation  8  (accounting  for  3  %  of  the  variance).  The  final  regression 
analysis  on  outcome  variables  revealed  that  for  the  Critical  Thinking  variable,  four  input 
variables  were  predictive  of  perceived  high  competence.  Expectation  3,  “I  would  meet 
other  people  and  learn  about  other  career  fields,”  entered  the  equation  first,  accounting  for 
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four  percent  (4  %)  of  the  variance.  The  Support  career  field  (information  management, 
personnel,  finance,  etc.)  also  was  predictive  of  high  perceived  competence  on  the  Critical 
Thinking  outcome.  However,  the  Support  career  field  had  an  inverse  relationship  with 
outcomes,  meaning  that  those  not  in  this  career  field  were  more  likely  to  score  themselves 
highly  competent  on  Critical  Thinking.  In  other  words,  the  Support  career  field  had  a 
negative  effect  on  students’  perceptions  of  high  competence  on  Critical  Thinking.  It 
entered  the  equation  at  step  2  and  accounted  for  two  percent  (2  %)  of  the  variance  over 
what  was  attributed  to  Expectation  3.  Age  2  and  Expectation  8  also  had  some  predictive 
power  for  the  Critical  Thinking  variable  but  after  Expectation  3  and  Support  entered  the 
equation,  not  much  was  left  for  these  two  variables  (1  %  and  2  %  respectively).  Overall, 
input  variables  accounted  for  the  most  variance  in  the  Critical  Thinking  variable  (9  %), 
with  the  least  variance  attributable  to  input  variables  occurring  in  the  Command  and 
Leadership  variable  (3  %).  These  data  are  valuable  in  determining  which  input  variables 
are  most  associated  with  high  perceived  competence  in  outcome  variables.  Although  it  is 
doubtful  that  key  leaders  at  ACSC  can  change  student  inputs,  by  knowing  this  information 
they  can  address  problem  areas  and  better  meet  the  needs  of  their  students. 

The  Effect  of  Environment  on  Outcomes.  The  discussion  thus  fer  has  centered  on 
demographics  and  student  expectations,  which  were  input  variables.  Obviously  these  were 
important  because  it  gave  the  researcher  insight  into  student  characteristics  which  may  be 
important  in  making  recommendations  about  the  program.  However,  the  College  has  little 
choice  as  to  the  type  of  student  who  comes  to  ACSC.  Students  are  chosen  as  a  result  of  a 
central  board  and  the  criteria  are  based  on  their  rank  as  well  as  their  abilities  to  hold  key 
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field  grade  (Major  and  above)  positions  in  the  future.  While  decision-makers  at  the 
College  do  not  have  control  over  student  characteristics  or  expectations,  they  do  have 
control  over  their  teaching  methods  and  program  activities.  The  results  of  the 
correlational  data  analysis  between  environmental  variables  and  outcomes  revealed  that 
the  Curriculum  variable  was  most  associated  with  outcomes,  with  Teaching  Methods 
following  Curriculum  closely.  The  largest  relationship  occurred  in  the  Joint  Campaign 
outcome  variable  with  Curriculum  (r  =.46),  while  the  smallest  significant  relationship  was 
between  Research  and  Critical  Thinking  (r=.13).  Surprisingly,  there  was  no  relationship 
between  Research  and  the  Air/Space  Power  outcome  variable.  One  can  conclude  fi'om 
these  data  that  Curriculum  items  were  most  associated  with  high  competence  on  outcome 
variables,  especially  in  Joint  Campaign.  On  the  other  end  of  the  spectrum,  the  Research 
component  of  the  College  was  least  associated  with  outcomes,  especially  Critical 
Thinking.  The  fact  that  Research  was  not  associated  with  the  Air/Space  Power  outcome 
variable  was  notable.  Apparently,  the  Research  component  of  the  College,  which  was  not 
even  associated  with  Air/Space  Power,  the  outcome  on  which  a  larger  percentage  of 
students  tended  to  score  themselves  as  highly  competent,  was  consistently  rated  as  being 
of  low  quality,  relative  to  the  other  variables.  These  data  seem  to  imply  that  ACSC 
Curriculum  items  are  the  most  facilitative  toward  students’  perceptions  of  high 
competence  on  outcomes,  while  Research  items  are  not.  Even  though  Curriculum  and 
Research  emerged  as  extremes,  the  other  variables,  especially  Teaching  Methods  and 
Faculty  were  also  significantly  associated  with  outcomes  and  should  not  be  discounted  as 
unimportant. 
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The  Effect  of  Environment  on  Outcomes  After  Controlling  for  Inputs.  Results  of  the 
predictive  analysis  substantiated  the  correlational  findings.  The  multiple  regression 
analysis  allowed  the  researcher  to  control  for  the  effects  of  input  variables,  while 
examining  the  effects  of  environmental  variables  on  outcomes.  Although  many 
researchers  use  coirelational  data  to  predict  or  show  causation,  multiple  regression  is  a 
more  powerful  statistical  tool  for  that  purpose.  In  this  case,  after  controlling  for  input 
variables,  Curriculinn  had  the  most  predictive  power  in  the  regression  equations  for  all 
four  outcome  variables  (ranging  fi’om  10  %  in  Command  and  Leadership  to  15  %  in  Joint 
Campaign).  Teaching  Methods  also  had  some  predictive  power  for  outcomes  above  what 
was  explained  by  Curriculum  (ranging  fi-om  2  %  in  Air/Space  Power  and  Critical  Thinking 
to  3  %  in  Joint  Campaign)  except  in  Command  and  Leadership,  where  Faculty  followed 
Curriculum  as  having  some  predictive  power  (4  %  after  Curriculum).  Overall,  the  amount 
of  variance  explained  by  environmental  variables  after  accounting  for  inputs  ranged  from 
fourteen  percent  (14  %)  in  Command  and  Leadership  and  Critical  Thinking  to  eighteen 
percent  (18  %)  in  Joint  Campaign.  This  suggests  that  after  controlling  for  inputs, 
respondents  who  rated  the  quality  of  Curriculum  as  high  were  most  likely  to  rate  their 
competence  in  outcomes  as  high.  After  Curriculum,  Teaching  Methods  emerged  as 
adding  some  variance  to  the  equation.  Because  most  of  the  variance  was  attributed  to 
Curriculum  and  Teaching  Methods,  there  was  no  predictive  power  left  for  the  other 
environmental  variables  such  as  Faculty,  Technology,  Grading,  and  Research.  This  is  not 
to  say  that  these  variables  should  be  ignored.  In  fact,  if  one  were  to  look  at  the  correlation 
coefficients  on  these  variables  as  outlined  in  table  38,  one  would  see  that  they  were  aU 
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significant  at  the  .01  level  except  for  Research  and  Air/Space  Power  Applications. 
Similarly,  when  all  environmental  variables  were  entered  into  the  regression  equations 
individually  after  inputs,  they  all  accounted  for  some  portion  of  the  variance.  Yet,  the  fact 
remains  that  Curriculum  emerged  as  the  most  predictive  environmental  variable  which 
suggests  that  this  element  of  the  program  is  most  fecilitative  toward  students’  perceptions 
of  high  competence  on  outcomes.  Conversely,  Research  items  consistently  appear  as  the 
least  associated  with  outcomes,  both  in  the  correlational  analysis  and  the  regression.  This 
suggests  that  this  element  of  the  program  is  not  as  successful  as  the  others  and  should  be 
targeted  for  possible  improvement  and  modification. 

Because  Curriculum  items  were  most  associated  with  and  most  predictive  of  outcome 
variables,  a  more  detailed  examination  of  individual  Curriculum  items  was  warranted.  The 
five  Curriculum  items  used  for  the  analyses  were: 

•  Item  59,  “Assigned  books.” 

•  Item  60,  “Curriculum  flow.” 

•  Item  6 1 ,  “Systems  approach  to  problem  solving.” 

•  Item  62,  “Balance  between  academics  and  social  functions.” 

•  Item  67,  “To  what  extent  do  you  think  ACSC  is  a  graduate-level 
program?’ 

Regression  analyses  of  individual  Curriculum  items  revealed  that  item  61,  “the  systems 
approach  to  problem  solving,”  and  item  59,  “assigned  books,”  had  the  most  predictive 
powers  on  outcomes.  This  suggests  that  students  who  rated  the  quality  of  the  systems 
approach  to  problem  solving  and  assigned  books  as  high  were  most  likely  to  rate  their 
competence  in  outcome  variables  as  high.  One  might  conclude  from  these  data  that  all 
that  needs  to  be  done  to  ensure  success  at  ACSC  is  to  concentrate  on  Curriculum  items. 
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particularly  the  systems  approach  and  assigned  books.  Undoubtedly,  a  strategy  such  as 
this  would  yield  similar  results  to  what  was  found  in  this  study. 

Descriptive  data  and  qualitative  results  added  details  to  this  overall  analysis.  When 
descriptive  statistics  on  Curriculum  were  compared,  “assigned  books”  stood  out  as  being 
the  most  highly  rated  item,  with  over  a  quarter  of  respondents  rating  the  books  as 
“outstanding.”  Respondents  also  rated  the  “systems  approach  to  problem  solving  as  high” 
but  below  “assigned  books.”  Although  students  rated  the  books  highly,  they  thought  that 
the  amount  of  assigned  reading  was  excessive.  Written  comments  from  the  open-ended 
section  of  the  questionnaire  also  discussed  this  problem.  Respondents  believed  that  while 
the  books  were  “outstanding,”  the  amount  of  reading  they  were  expected  to  accomplish 
did  not  allow  them  enough  time  for  synthesis  or  analysis.  Descriptive  data  and  qualitative 
comments  on  Research  also  validated  the  predictive  data.  Group  research  was  one  of  the 
lowest  rated  items  on  the  questionnaire  (grading  procedures  was  the  lowest). 

Respondents  were  disappointed  that  they  had  not  received  adequate  training  or  support 
for  their  research  projects.  Their  frustrations  seemed  to  center  on  the  incongruence 
between  the  lack  of  support  for  research  from  faculty  and  staff  while  the  College  placed  a 
great  deal  of  emphasis  on  that  portion  of  the  program.  Grading  procedures  were  also  not 
rated  very  high.  This  seemed  to  also  revolve  around  research  projects.  In  written 
comments,  students  wrote  that  the  lack  of  standardization  in  grading  kept  many  people 
from  being  chosen  as  distinguished  graduates.  Many  also  wrote  that  by  having  assigned 
grades,  the  competitive  nature  of  most  people  detracted  from  the  learning  experience. 
Descriptive  data  and  qualitative  open-ended  comments  from  the  questionnaire  seemed  to 
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suggest  the  same  conclusions  derived  from  the  quantitative  analyses.  However,  another 
item  stood  out  in  respondents’  written  comments  that  should  be  added  here.  Graduates  of 
ACSC  were  very  critical  of  military  feculty  members  (ahhough  no  names  were 
mentioned).  According  to  their  comments,  they  believed  that  military  faculty  were  not 
qualified  to  teach  them.  An  element  of  bitterness  showed  through  in  many  comments  in 
which  respondents  wrote  that  the  school  was  allowing  oflBcers  to  come  to  ACSC  to  teach 
even  though  they  had  not  been  “good  enough”  to  get  selected  to  attend  the  school 
themselves.  Whether  or  not  this  was  a  valid  observation,  the  fact  remains  that  there  seems 
to  be  a  consensus  among  graduates  that  the  military  feculty  members  were  deficient. 
Quantitative  data  seemed  to  corroborate  these  sentiments  as  the  data  show  that  students 
rated  civilian  faculty  members  as  being  of  higher  quality  than  military  fiiculty  members. 

The  conclusions  highlighted  above  only  touched  the  surfece  Of  the  findings  of  this 
study.  However,  the  conclusions  chosen  for  inclusion  in  this  chapter  represent,  in  a  broad 
sense,  the  most  important,  overarching  findings.  The  ptupose  of  this  study  was  to  develop 
a  comprehensive  institution- wide  assessment  program  for  Air  Command  and  Staff 
College.  Comprehensibility,  by  its  very  nature,  sacrifices  depth  for  breadth.  By  looking  at 
broad  brush  strokes,  one  sacrifices  many  niiances  that  occur  as  a  result  of  any  assessment. 
ACSC  is,  by  higher  education  standards,  a  short  course  and  assessment  efforts  must  be 
quick,  comprehensive,  and  readable  but  they  also  must  provide  information  that  leads  to 
program  improvement.  Decision-makers  at  ACSC  have  shown  that  they  are  willing  to 
make  dramatic  changes  in  their  curriculum.  This  study  represents  a  place  for  them  to  start 
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when  they  examine  what  they  are  currently  doing.  It  answered  questions  about  what 
worked  and  what  did  not  work  quite  as  well,  at  least  as  perceived  by  graduates  of  the 
program.  Although  the  study  does  have  several  limitations  (these  will  be  discussed  in 
detail  later),  it  represented  one  way  in  which  targeted,  detailed  information  about  program 
elements  could  be  gathered.  It  also  showed  how  the  data  could  be  examined  and  reported. 
Clearly,  there  is  still  much  that  needs  to  be  answered  but  the  researcher  believes  that  this 
study  provided  a  place  in  which  one  could  start  investigating  a  short,  yet  complicated 
program  such  as  Air  Command  and  Staff  College. 

Recommendations  and  Implications 

The  final  question  (#7)  posed  in  chapters  one  and  three  of  this  dissertation  was: 
“What  were  the  implications  of  the  results  of  this  study  for  Air  Command  and  Staff 
College?”  This  section  addresses  both  recommendations  and  uiq)lications. 
Recommendations  are  based  on  the  results  of  the  data  analyses  and  the  conclusions 
already  discussed.  Implications  refer  to  policy  and  research  implied  by  the  results. 

The  first  recommendation  for  decision-makers  at  ACSC  is  to  look  at  the  mission 
statement.  While  the  current  mission  statement  explicitly  states  the  focus  of  the  College, 
it  is  obvious  that  the  implied  goals  of  the  program  do  not  deal  only  with  Air/Space  Power 
Applications.  Command  and  Leadership  and  Critical  Thinking,  not  to  mention  Joint 
Campaign  all  emerged  as  important  aspects  of  ACSC.  The  mission  statement  should  be 
concise  but  it  should  also  encompass  all  of  the  major  goals  of  the  program.  The  data 
implied  that  Command  and  Leadership  was  an  area  in  which  students  felt  the  least 
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competent.  This  does  not  necessarily  mean  that  students  are  not  learning  Command  and 
Leadership  at  ACSC  as  the  data  showed  that  students’  perceptions  of  competence  on  all 
outcomes  were  relatively  high.  However,  ACSC  leaders  may  want  to  take  a  look  at  how 
they  are  teaching  the  Command  and  Leadership  outcome  and  give  some  thought  to  ways 
in  which  they  might  improve  facilitating  this  skill.  Although  the  quantitative  data 
revealed  that  students  perceived  the  most  competence  in  Air/Space  Power,  written 
comments  suggested  that  some  students  were  finstrated  because  the  school  wanted  to 
mold  them  into  air  campaign  planners  and  many  did  not  see  the  utility  in  learning  this  skill. 
Of  course,  focusing  on  weak  or  strong  areas  are  not  mutually  exclusive.  The  College 
could  have  a  strong  air  campaign  element  and  a  strong  Command  and  Leadership 
component  as  well  which  would  address  both  areas.  Perhaps  Command  and  Leadership 
skills  could  be  incorporated  “across  the  curriculum”  rather  than  only  taught  in  a  module 
set  aside  for  that  purpose.  Students  could  be  learning  how  to  be  air  campaigners  and 
commanders  at  the  same  time.  This  would  require  some  creative  pedagogy  across  aU 
courses  as  well  as  collaborative  teaching  techniques  between  faculty  members,  but  it 
would  be  a  worthwhile  experiment  and  one  in  which  everyone  would  benefit. 

This  study  did  not  administer  any  type  of  pretest  to  students  prior  to  their  exposure  to 
the  learning  environment  at  ACSC  because  of  time  constraints.  However,  a  second 
recommendation  for  the  College  would  be  to  give  students  a  pretest  on  the  major 
components  of  the  program.  The  College  could  devise  its  own  instruments  for  measuring 
students’  abilities  as  they  come  into  the  program.  They  could  also  use  standardized 
instruments  devised  and  validated  by  experts,  especially  when  measuring  skills  such  as 
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critical  thinking.  Pretest  data  on  incoming  students  would  prove  to  be  helpful  for 
discovering  strengths  and  weaknesses  of  each  class.  Longitudinal  data  could  be  compiled 
and  patterns  studied  for  future  improvement  efforts  at  the  College.  Pretest  data  could  also 
become  part  of  a  larger  assessment  program  in  which  different  teaching  techniques  and 
program  activities  are  implemented  and  tested.  Assessment  data  should  be  tied  to  student 
talent  development  and  the  College’s  programs  continually  reviewed  and  improved  as  a 
result. 

According  to  the  correlational  and  predictive  analyses,  Curriculum  items  were  the  most 
associated  with  outcomes.  This  could  suggest  two  things:  students’  perceptions  of  high 
competence  existed  prior  to  their  arrival  at  ACSC  or  the  College  did  a  good  job  of 
communicating  the  Curriculum  items  to  students.  Whatever  the  case,  the  school  must 
decide  if  they  wish  to  reinforce  program  elements  on  which  students  already  perceive  high 
competence  or  if  they  are  interested  in  focusing  on  elements  on  which  students’  perceived 
themselves  less  competent.  Of  aU  the  Curriculum  items,  item  61,  “the  systems  approach 
to  problem  solving,”  and  item  59,  “assigned  books”  were  the  most  associated  with 
outcomes.  In  other  words,  those  who  rated  these  program  elements  highly  were  also 
those  who  perceived  greater  competency  on  outcomes.  If  ACSC  could  get  students  to  see 
the  “systems  approach”  and  “assigned  books”  as  more  important,  then  higher  outcomes 
might  foDow.  Even  though  students  rated  the  quality  of  the  assigned  books  highly,  they 
repeatedly  criticized  the  reading  load  in  their  written  comments.  Students  perceived  that 
not  enough  time  was  allotted  for  them  to  discuss  the  readings  because  the  load  was  too 
unrealistic.  Apparently,  ACSC’s  goal  is  not  only  to  provide  students  with  reading 
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opportunities,  but  also  to  emphasize  research  and  critical  thinking  skills.  Reading  for 
reading’s  sake  will  not  create  an  environment  that  fecilitates  research  or  critical  thinking. 
While  students  should  be  exposed  to  various  viewpoints,  historical  examples,  and  current 
events,  they  should  also  have  a  chance  to  synthesize  and  analyze.  To  this  end,  students 
should  be  given  reading  assignments  that  provide  them  with  background  information  on  an 
issue,  but  they  should  be  encouraged  to  form  their  own  opinions  about  it  and  be  allowed 
to  discuss  what  they  read  with  others.  The  recommendation  in  this  area  would  be  to 
assign  less  reading,  emphasize  research,  and  facilitate  discussions  and  interaction  between 
students. 

The  Research  element  of  the  College  was  consistently  one  of  the  most  criticized  and 
lowest  rated  elements  of  the  overall  program.  The  relational  and  predictive  analyses 
revealed  that  Research  was  the  least  associated  with  outcomes  and  one  of  the  least 
predictive  elements  of  the  school.  This  indicates  that  decision-makers  at  ACSC  need  to 
take  a  hard  look  at  the  Research  program  and  decide  what  their  goals  should  be  in  this 
area.  Is  ACSC  a  “research”  institution?  Is  it  important  to  teach  mid-career  officers  to  be 
researchers?  This  study  revealed  that  students  were  most  frustrated  with  group  research 
grades  and  the  lack  of  direction  on  how  to  do  research.  The  College  caimot  assume  that 
all  of  their  students  have  had  adequate  training  on  research  skills  or  that  they  even  know 
how  to  work  on  team  projects.  It  is  important  for  those  who  are  involved  in  grading 
student  research  projects  to  have  an  opportunity  to  interact  with  students.  Training 
should  be  available  for  those  who  need  it  and  faculty  mentors  should  be  assigned  to 
research  teams  to  help  facilitate  the  process.  Grading  procedures  must  be  outlined  in 
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detail  and  students  made  aware  of  what  is  expected  of  them.  These  are  just  a  few 
recommendations  that  could  help  alleviate  some  of  the  finstration  about  the  research 
process.  If  ACSC  is  serious  about  teaching  research  skills  and  promoting  excellent 
research  projects,  then  students  should  be  given  ample  time  to  pursue  this  element  of  the 
program  and  they  should  be  given  constant  mentoring  by  those  who  are  qualified.  In 
addition,  an  element  of  “mastery”  learning  should  be  built  into  the  research  program. 
Students  should  work  with  a  faculty  mentor  and  be  allowed  to  continually  rewrite  and 
revise  their  research  products.  Students  should  be  allowed  to  choose  their  own  research 
projects,  perhaps  even  submitting  proposals  for  what  they  want  to  do.  Ideas  fi'om  around 
the  Air  Force  could  be  solicited  and  “real-world”  problems  could  be  addressed.  This 
method  works  weU  in  business  and  industry  and  in  some  highly  successful  business  schools 
(Mann  and  Staudenmier,  1991).  By  providing  students  with  an  opportunity  to  work  on 
issues  and  problems  that  commanders  and  senior  Air  Force  leaders  identify  as  central, 
ACSC  would  be  providing  a  service  to  the  Air  Force  while  providing  students  with 
considerable  learning  experiences.  Finally,  group  research  is  a  good  idea  as  Air  Force 
ofBcers  must  often  work  coUaboratively  on  projects.  This  study  revealed  that  students’ 
perceptions  of  the  Research  element  at  ACSC  was  relatively  negative  and  they  voiced, 
through  their  written  comments,  that  they  were  fiaistrated  with  the  research  process  at 
ACSC  because  they  were  not  given  adequate  training  on  research  or  provided  ample  time 
to  work  on  their  projects.  Therefore,  it  is  important  that  student  groups  not  be  left  to 
their  own  devices  while  conducting  their  research.  They  should  be  guided  on  both  group 
processes  and  research  principles  by  an  experienced  faculty  member.  Again,  if  ACSC  is 
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serious  about  providing  a  learning  environment  that  emphasizes  research,  decision  makers 
must  be  committed  to  it  and  provide  ample  support  for  both  faculty  and  students. 

While  the  Curriculum  and  Research  variables  represented  both  ends  of  the  spectrum 
after  all  elements  of  the  program  were  examined,  the  other  environmental  variables 
(Teaching  Methods,  Technology,  Faculty,  and  Grading)  were  also  significantly  associated 
with  outcomes  and  should  not  be  overlooked.  Teaching  Methods,  next  to  Curriculum, 
was  most  associated  with  the  outcomes  and,  in  most  cases,  emerged  as  providing  some 
predictive  power  after  inputs  and  Curriculum  entered  the  regression  equations.  A  notable 
observation  about  students’  perceptions  of  Teaching  Methods  was  that  students  rated  the 
quality  of  lectures  by  guest  lecturers  as  the  highest  among  all  the  methods  used  with  the 
lowest  rated  method  being  lectures  by  instructors.  This  seems  to  imply  that  students 
prefer  to  be  taught  by  guest  lecturers  rather  than  ACSC  faculty  members.  Students  did 
rate  the  quality  of  military  feculty  lower  than  civilian  fecuhy  and  criticized  them  in  their 
written  comments.  Senior  leaders  at  ACSC  have  already  taken  steps  to  help  improve  the 
quality  of  their  military  faculty  members  by  obtaining  sponsored  Ph.D.  slots  for  selected 
faculty  members.  While  this  study  did  not  indicate  that  students  might  rate  the  feculty 
higher  if  they  had  Ph.D.s,  it  is  still  a  step  in  the  right  direction. 

.  Technology  is  a  dynamic  and  necessary  tool  which  facilitates  communication  and 
creates  numerous  possibilities  for  education  and  research.  ACSC  has  done  a  tremendous 
job  in  a  short  period  of  time  by  providing  students  with  the  hardware  and  by  giving  them 
an  overview  of  the  uses  of  technology  in  a  classroom  setting.  However,  the  school  must 
not  assume  that  everyone  comes  to  the  school  with  the  same  knowledge  about  the 
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internet,  the  worldwide  web,  or  even  basic  word  processing.  Training  is  essential  and 
constant  reinforcement  about  how  to  use  technology  is  a  must.  Not  only  should  students 
be  learning  about  the  internet,  the  worldwide  web,  e-mail,  and  Toolbook©,  but  they 
should  be  exposed  to  many  aspects  and  possibilities.  Some  students  may  want  to  know 
how  to  use  a  database  program  like  “Excel”  or  a  statistical  package  such  as  “Statistical 
Package  for  the  Social  Sciences.”  Others  may  be  interested  in  learning  how  to  use 
presentation  programs  like  “Powerpoint.”  Other  interactive  software  in  addition  to 
Toolbook©  also  exist  that  might  appeal  to  some  students.  While  recognizing  the  benefits 
of  technology  in  the  classroom,  one  must  also  recognize  the  downside.  The  data  revealed 
that  students  rated  the  quality  of  computerized  readings  as  lowest  among  all  Technology 
items  (see  table  5).  Placing  reading  assignments  on  the  net  should  be  avoided  unless 
students  have  access  to  a  printer.  Interactive  programs  such  as  Toolbook©  can  enliven 
text,  but  they  should  not  replace  the  written  word.  The  pros  and  cons  of  the  various  uses 
of  technology  should  be  investigated  and  different  techniques  tried  and  tested.  Finally,  the 
pace  of  technology  moves  so  quickly  that  imless  one  devotes  time  to  it  constantly,  new 
and  exciting  software  and  uses  for  them  can  be  easily  overlooked.  If  ACSC  is  committed 
to  how  technology  can  be  used  in  their  program,  and  it  appears  as  if  they  are,  they  should 
hire  a  technologist  to  keep  abreast  of  the  latest  changes  and  to  train  the  feculty  and  staff 
on  its  uses.  This  individual  could  be  someone  who  is  already  at  the  school  or  they  could 
invest  additional  fimds  into  hiring  someone  who  knows  and  understands  the  complexities 
that  surroimd  the  use  of  technology. 
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As  already  stated,  the  purpose  of  asking  students  to  rate  the  quality  of  the  feculty  was 
to  look  at  their  perceptions  about  the  balance  between  military  and  civilian  feculty 
members  and  to  find  out  to  what  extent  they  thought  the  feculty  should  consist  of  more 
Ph.D.s.  The  results  of  the  data  analyses  revealed  that  students  rated  the  quality  of  civilian 
faculty  members  higher  than  military  feculty  but  they  believed  that  the  balance  between  the 
two  was  “about  right.”  When  asked  to  what  extent  they  thought  the  ACSC  faculty  should 
consist  of  more  Ph.D.s,  over  half  of  respondents  replied  in  the  “very  little”  and  “to  some 
extent”  categories.  Written  comments  in  the  open-ended  section  of  the  questionnaire 
added  to  this  quantitative  analysis.  Students  criticized  the  military  faculty  members  for  not 
being  qualified  to  teach.  In  other  words,  respondents  did  not  believe  that  military  faculty 
members  were  “subject  matter”  experts.  It  is  difficult  to  find  people  throughout  the 
military  that  are  both  qualified  to  teach  and  who  volunteer  to  come  to  ACSC  to  be  on 
faculty.  This  issue  has  been  a  continual  problem  for  Air  Force  PME  and  is  not  one  that  is 
likely  to  be  alleviated  soon  (Davis  and  Donnini,  1991).  As  was  already  stated,  ACSC 
senior  leaders  have  invested  a  lot  of  time  and  resources  into  obtaining  sponsorship  to  send 
some  faculty  members  to  obtain  Ph.D.s.  While  having  more  Ph.D.s  on  faculty  will 
improve  the  chance  that  many  will  be  subject  matter  experts  and  know  how  to  conduct 
and  direct  research,  one  must  remember  that  rank  and  experience  probably  count  for  more 
than  advanced  degrees  in  a  setting  like  ACSC.  Faculty  members  with  new  Ph.D.s  may  be 
able  to  teach  history  or  political  science  issues,  and  they  may  be  more  qualified  to  teach 
students  how  to  proceed  on  their  research  projects,  but  will  they  have  the  kind  of  field 
experiences  that  add  credibility  to  what  they  are  saying?  This  issue  could  be  alleviated 
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somewhat  by  bringing  in  people  with  the  necessary  experience.  The  guest  lecturer 
program  is  one  avenue  but  students  appear  to  want  to  be  able  to  interact  with  those  who 
have  firsthand  experience.  Again,  ACSC  must  decide  what  the  focus  of  the  College 
should  be.  If  they  want  the  school  to  be  an  academic  program,  complete  with  a  number  of 
Ph.D.s  in  history,  political  science,  and  social  science,  then  it  is  appropriate  to  infuse  the 
faculty  with  more  people  with  advanced  degrees.  Conversely,  if  ACSC  wants  to 
concentrate  on  military  issues,  particularly  air  canqjaign  planning,  then  perhaps  it  is  more 
appropriate  to  hire  more  officers  with  current  field  experience.  These  officers  do  not 
necessarily  have  to  be  all  pilots.  In  fact,  given  the  feet  that  there  are  more  non-rated 
officers  attending  the  program,  it  would  be  inappropriate  to  hire  only  pilots.  To  attract  the 
most  qualified  feculty  members,  ACSC  should  be  committed  to  their  faculty  by  providing 
them  with  time  to  be  innovative  and  creative.  They  should  be  recognized  for  their  hard 
work  and  encouraged  to  pursue  their  own  research  projects  and  to  present  them  at 
national  conferences.  Faculty  development  programs  should  be  an  important  part  of  the 
College  and  perhaps  centrally  directed  and  administered  by  someone  whose  prime 
responsibility  is  to  help  feculty  members  in  all  aspects  of  their  job. 

The  lowest  rated  item  on  the  questionnaire  was  grading  procedures,  with  nearly  fifty- 
two  percent  (51.6  %)  of  respondents  rating  it  as  “unsatisfactory”  or  “marginal.”  Written 
comments  corroborated  this  data.  Obviously,  grading  procedures  at  ACSC  were 
perceived  as  less  than  adequate.  Of  particular  concern  for  students  was  the  way  in  which 
grades  were  used  for  determining  distinguished  graduates.  Most  respondents  who  wrote 
about  this  topic  recommended  that  the  DG  program  be  discontinued.  While  this  is 
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unlikely  to  happen,  it  is  important  for  ACSC  to  look  at  how  grades  are  being  assigned  and 
how  the  perception  that  grades  are  assigned  arbitrarily  can  be  changed.  In  addition, 
decision-makers  at  the  College  need  to  decide  if  they  perceive  the  school  to  be  a  graduate 
program  or  if  they  are  mostly  interested  in  providing  management  development  education 
in  the  tradition  of  business  and  industry.  If  the  school  is  a  graduate  program,  then  grades 
are  appropriate  and  students  should  be  working  toward  a  Master  Degree.  If  it  is  a 
management  development  program,  then  grades  are  inappropriate  and  the  focus  of  the 
school  should  be  on  team  building  and  real-world  research  projects.  These  are  important 
distinctions  but  the  current  environment  at  the  school  suggests  that  they  are  trying  to  do 
both.  After  the  College  defines  what  it  is  all  about,  it  must  communicate  this  to  the 
students  while  letting  them  know  what  is  expected  of  them. 

The  conclusions  and  recommendations  highlighted  in  this  chapter  represent  the  broad 
themes  implied  by  the  data  analyses.  Some  of  the  recommendations  can  be  easily 
implemented  while  others  may  take  a  change  in  policy  and  a  large  commitment  by  those  in 
key  positions.  Reviewing  the  mission  statement  and  defining  what  direction  the  College 
will  take  in  the  future  will  be  the  first  challenge.  The  results  of  the  data  brought  out  in 
this  study  revealed  that  ACSC  is  doing  many  things  well  and  that  the  needs  of  a  majority 
of  their  students  are  being  met.  On  the  average,  students  perceived  themselves  as  being 
highly  competent  on  all  four  outcomes,  but  there  were  areas  in  which  students  did  not 
score  themselves  as  highly  competent  as  others.  Generally  speaking,  students  also  rated 
the  program  activities  and  teaching  methods  used  at  ACSC  to  be  of  high  quality.  Yet, 
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some  elements  of  the  program  were  not  perceived  as  being  as  effective  as  others  and 
warrant  further  inspection.  This  chapter  outlined  some  of  these  elements  and  made 
recommendations  about  what  could  be  done  to  remedy  problem  areas. 

This  study  did  not  deal  with  ACSC’s  impact  on  the  Air  Force  or  the  role  of 
professional  military  education  in  the  modem  military.  However,  the  data  revealed  that 
some  elements  of  the  program,  particularly  the  research  element,  lent  themselves  to 
addressing  real-world  problems  and  issues.  While  students  are  exposed  to  current  issues 
and  problems,  they  do  not  seem  to  be  part  of  the  solutions.  It  would  be  advisable  for  key 
leaders  at  ACSC  to  find  out  what  problems  the  various  commands  face  in  their  day-to-day 
operations.  Research  at  ACSC  should  target  real-world  problems  in  which  groups 
formulate  solutions  and  present  them  to  commanders  throughout  the  Air  Force  (or  Army, 
Navy,  Marines).  Students  should  be  encouraged  to  work  on  problems  that  they  find 
engaging  and  that  apply  to  their  particular  expertise.  By  having  students  work  on  research 
projects  that  address  current  issues  and  problems,  students  will  see  relevance  in  what  they 
are  doing  and  the  College  can  be  assured  that  their  students  are  also  learning  how  to  be 
critical  thinkers  and  competent  researchers.  In  addition,  ACSC  will  become  the  place  in 
the  Air  Force  where  theory,  research,  and  solutions  come  together  in  a  student-centered 
learning  environment. 


Limitations  and  Future  Research 

Limitations  of  this  study  were  already  discussed  in  detail  in  chapter  three.  The  four 


major  limitations  outlined  involved  the  scope  of  the  study,  design  limitations,  self- 
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reporting,  and  the  homogeneity  of  the  study  group.  Out  of  those  four,  the  scope  of  the 
study  and  design  limitations  are  the  most  serious.  When  design  decisions  were  made  for 
this  dissertation,  the  researcher  had  several  constraints,  time  and  proximity  to  the  College 
being  the  most  detrimental.  While  the  best  approach  may  have  been  to  give  students  a 
pretest,  follow  their  progress  throughout  the  ten  months,  and  end  with  a  series  of 
posttests,  it  was  not  possible  to  implement  such  a  plan.  However,  this  should  be  done  in 
future  research  activities.  A  longitudinal  study  of  one  class  should  be  designed  and 
implemented.  Additionally,  several  longitudinal  studies  should  be  accomplished  and  a  data 
base  of  information  compiled  and  change  over  time  studied  and  patterns  reported. 
Quantitative  data  should  be  supplemented  by  qualitative  information  compiled  through 
observations  and  interviews.  These  data  could  then  be  compared  to  quantitative  statistics 
to  reinforce  findings  and  recommendations.  Major  decisions  should  not  be  made  without 
several  reinforcing  studies  suggesting  similar  directions.  In  fact,  the  school  lends  itself  to 
becoming  a  laboratory  for  research  with  unlimited  possibilities.  The  possibilities  at  ACSC 
are  endless  with  a  new  class  coming  in  every  year.  Faculty  and  staff  at  the  College  should 
be  encouraged  to  design  and  implement  experiments  that  will  help  them  decide  what 
methods  work  and  what  does  not  work.  For  example,  two  sections  could  be  taught  the 
same  material  using  different  techniques,  perhaps  the  case  study  method  in  one  instance 
and  colloquia-style  seminars  in  another.  An  ANOVA  design  for  an  experiment  such  as 
this  would  yield  quantitative  information  about  which  method  was  the  most  effective. 

This  could  be  done  for  any  number  of  program  activities  used  at  ACSC. 
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This  study  was  interested  in  developing  a  comprehensive  assessment  program  that 
could  be  implemented  quickly  and  findings  reported  in  the  shortest  possible  time.  As  a 
result,  the  researcher  sacrificed  depth  for  breadth.  This  study  presented  broad  patterns, 
but  future  studies  should  target  one  area  of  the  program  and  design  a  study  that  looks  at 
only  that  area.  For  example,  the  technology  component  of  the  College  has  become  very 
important  across  the  curriculum  and  lends  itself  to  being  the  subject  of  several  future 
studies.  Any  number  of  ACSC  program  elements  could  be  investigated  for  their  plausible 
uses  and  limitations. 

In  describing  the  limitations  and  implications  for  future  research,  one  runs  the  risk  of 
demeaning  one’s  own  research.  Yet,  research  is  always  an  ongoing  process  with  each 
study  revealing  areas  that  should  be  investigated  further.  While  most  studies  ask  more 
questions  than  they  answer,  one  of  the  purposes  of  doing  research  is  to  discover  areas  of 
concern  and  to  hold  them  up  to  the  light  for  examination.  This  study  was  able  to  do  just 
that,  the  limitations  notwithstanding.  There  is  still  much  that  needs  to  be  done  with  the 
assessment  program  at  ACSC.  It  should  evolve  continually,  asking  for  deeper  and  more 
detailed  data.  The  survey  instruments  should  be  constantly  reviewed  and  revised.  Yet, 
research  for  research’s  sake  is  not  necessarily  a  productive  activity.  Researchers  at 
ACSC,  as  with  any  higher  education  institution,  should  tie  their  assessment  efforts  to  both 
accountability  and  improvement.  To  do  less  would  discredit  the  research  process  as  well 


as  those  who  are  served  by  it. 
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Survey  Code _ 

Student  Perceptions  of  Program 
Effectiveness 
of 

Air  Command  and  Staff  College 

Introduction:  This  questionnaire  is 
intended  to  gather  information 
about  the  effectiveness  of  Air 
Command  and  Staff  College 
program  activities  and  teaching 
methods  in  accomplishing  its 
primary  program  goals.  BE 
HONEST  in  your  evaluation  of  the 
program.  Your  individual 
responses  will  be  kept 
CONFIDENTIAL  and  the  data 
gathered  from  your  comments  will 
be  reported  in  aggregated  form 
only. 

General  Directions:  Sections  A,  B, 
and  C  are  preceded  by  a  rating 
scale.  Read  each  item  carefiilly 
and  place  the  number  that  best 
describes  your  perception  in  the 
space  provided.  The  last  portion 
of  section  C  and  section  D, 
demographic  data,  requires  you  to 
place  a  check  mark  next  to  the 
appropriate  response. 

A.  Overall  EfiTectiveness 

Specific  Directions:  Questions  M 
are  designed  to  determine  your 
overall  evaluation  of  your 
educational  experiences  at  Air 
Command  and  Staff  College. 
Please  rate  questions  1-3  using  the 
rating  scale  below  by  placing  the 
appropriate  number  in  the  space 
provided. 


l=StrongIy  Disagree 
2=Disagree  3=Neutral 
4= Agree  5=Strongly  Agree 


_ 1 .  Overall,  the  program  was 

effective. 

_ 2,  I  consider  this  program  a 

valuable  experience  in  my 
professional  career. 

_ 3.  I  would  recommend 

attendance  at  ACSC  to  other 
officers. 


Use  the  following  rating  scale  to 
answer  question  4: 


l=Unsatisfactory 
2=Marginal  3=Satisfactory 
4=Excellent  5=Outstandiiig 


_ 4.  The  mission  of  Air 

Command  and  Staff  College  is  ‘^o 
educate  mid-career  officers  to 
develop,  advance,  and  apply  air 
and  space  power  in  peace  and 
war.”  How  well  do  you  think  the 
program  accomplished  that 
mission? 

B.  Course  Objectives 

Specific  Directions:  The  following 
questions  are  designed  to  gather 
data  on  your  perceptions  about 
your  competency  in  a  number  of 
areas.  Please  rate  questions  5-49 
using  the  scale  below. 


l=No  competence 
2=Very  little  competence 
3=  Some  competence 
4^Considerable  competence 
5=Great  competence 


How  would  you  rate  your 
competency  on  the  following 
items  at  the  point  in  time  when 
you  completed  Air  Command 
and  Staff  College? 

_ 5.  Comparing  different 

personality  types  and  using  them 
to  make  decisions. 

_ 6.  Understanding  and 

applying  leadership  skills  in 
diverse  situations. 

_ 7.  Understanding  and 

applying  counseling  techniques  in 
a  variety  of  situations. 

_ 8.  Applying  principles  of 

oral  and  written  communication  in 
a  variety  of  situations. 

_ 9.  Understanding  the  Air 

Force  officer  promotion  system. 


_ 10.  Understanding  and 

applying  quality  force  tools  and 
techniques. 

_ 11.  Understanding  the 

special  nature  of  the  military 
leader, 

_ 12.  Understanding 

standards  of  conduct  and 
officership. 

_ 13.  Understanding  combat 

leader  styles  throughout  histor>^ 

_ 14.  Understanding  the 

inclusion  of  a  strong  moral 
character  and  ethical 
considerations  in  decision-making. 

_ 15.  Understanding 

commanders’  roles  and 
responsibilities. 

_ 16.  Understanding  key 

elements  of  the  Army,  Navy,  Air 
Force,  and  Marine  officer 
evaluation  systems. 

_ 17.  Challenging 

assumptions. 

_ 18.  Breaking  down  barriers 

to  creative  thinking. 

_ 19.  Translating  ideas  into 

action. 

_ 20.  Understanding  and 

implementing  change  in  a  military 
environment. 

_ ^21.  Understanding  the 

differences  between  conflict  and 
war. 

_ 22.  Understanding  the 

social  aspect  of  war. 

_ ^23.  Understanding  the 

circumstances  of  war. 

_ 24.  Understanding  and 

applying  war  and  conflict  concepts 
in  different  historical 
circumstances. 
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_ 25.  Understanding  and 

creating  war  theory. 

_ 26.  Understanding  the 

relationship  of  war  theory  to  the 
practice  of  waging  war. 

_ 27.  Understanding  the 

interrelationship  of  historical 
experience  and  the  development  of 
war  theory. 

_ 28,  Ability  to  critically 

analyze  past  and  present  war 
theory  concepts. 

_ 29.  Understanding  how  to 

influence  international  actors  to 
achieve  national  objectives, 

_ ^30.  Identifying  and 

analyzing  a  military  force’s  center 
of  gravity. 

_ ^31.  Understanding  the 

military  instrument  of  national 
power, 

_ ^32,  Synthesizing  new  ideas 

on  the  application  of  air  and  space 
power. 

_ ^33.  Analyzing  and  applying 

war  termination  concepts  and 
issues  in  current  and  potential 
scenarios. 

_ 34.  Understanding  the  role 

of  innovation  in  superior  war 
fighting  capability. 

_ ^35.  Understanding  the 

critical  factors  that  the 
Commander-in-Chief  needs  in 
order  to  plan  and  prosecute  a 
theater  military  operation. 

_ ^36.  Understanding  the 

structure  of  a  joint  organization. 

_ 37.  Understanding  the  role 

of  the  Air  Force  in  supporting  joint 
planning. 

_ 38.  Understanding  and 

applying  joint  operations  planning 
doctrine  and  procedures. 


_ 39.  Understanding  the  roles, 

functions,  capabilities  and 
limitations  of  the  US  military 
forces  that  affect  joint  and 
combined  operations. 


perceptions  of  the  quality  of 
program  activities  and  teaching 
methods  at  Air  Command  and 
Staff  College.  Please  rate 
questions  50-66  using  the  scale 
below. 


_ ^40.  Understanding  and 

applying  lessons  learned  fi*om 
classic  military  campaigns 
throu^out  history. 

_ ^41.  Understanding  the 

influence  of  national  policy  and 
strategy  on  the  joint  /combined 
planning  process. 

_ ^42.  Understanding  basic 

concepts  and  issues  in  plaiming 
and  executing  war  termination. 

_ ^43.  Understanding  the 

political,  economic,  and  military 
roles  in  post-hostilities 
environment. 

_ ^44.  Understanding  the  use 

of  technology  to  conduct  and  win 
campaigns. 

_ ^45.  Understanding  the  need 

for  resource  management  and 
acquisition  to  support  fixture  force 
development. 

_ 46.  Understanding  the  role 

and  impact  of  technology  on  air 
campaign  planning. 

_ 47.  Understanding  the 

contributions  of  air  and  space 
power  to  theater  campaigns. 

_ 48.  Understanding  and 

applying  the  air  campaign  process 
to  national  scenarios. 

_ 49.  Understanding  and 

analyzing  the  impact  of  contextual 
and  operational  art  elements  on 
military  campaigns. 

C.  Program  Activities  and 
Teaching  Methods 

Specific  Directions:  The  following 
questions  are  designed  to  rate  your 


l=Unsatisfactory 

2=Marginal 

3=Satisfactory 

4=£xcelleiit 

5=Out$tanding 

How  would  you  rate  the  quality 
of  the  following  program 
elements  at  Air  Command  and 
Staff  College? 

_ 50.  Lectures  by  course 

instructors. 

_ 51.  Lectures  by  guest 

lecturers. 

_ 52.  Informal  lecture 

seminars. 

_ 53.  Colloquia  style 

seminars. 

_ 54.  Use  of  computers  for 

daily  schedules, 

_ 55.  Use  of  computers  for 

testing. 

_ 56.  “Toolbook”  programs  to 

supplement  readings. 

_ 57.  Computerized  readings. 

_ 58.  Level  of  computer 

instruction  or  hands-on 
training. 

_ 59.  Assigned  books. 

_ 60.  Curriculum  flow. 

_ 61.  Systems  approach  to 

problem  solving, 

_ 62.  Balance  between 

academics  &  social 
fiinctions. 

_ 63.  Group  research  projects. 

_ 64.  Military  Faculty 

members. 

_ 65,  Civilian  Faculty 

members. 

_ 66.  Grading  procedures. 

Use  the  following  rating  scale  to 
answer  67-74. 


l=Not  at  all 

2=Very  little 

3=To  some  extent 

4=To  a  considerable  extent 

5=To  a  great  extent 
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_ 61,  To  what  extent  do  you 

think  ACSC  is  a  graduate-level 
program? 

_ ^68.  To  what  extent  were 

you  given  adequate  training  on 
how  to  do  research? 

_ 69,  To  what  extent  did  you 

receive  adequate  support 
(resources,  faculty  assistance)  with 
your  research  project? 

_ ^70.  To  what  extent  should 

the  ACSC  faculty  consist  of  more 
members  who  have  Ph.D.s? 

_ 71 .  To  what  extent  did  you 

receive  adequate  feedback  on  your 
oral  and  written  work? 

_ 72.  To  what  extent  did  you 

know  about  and  understand  the 
reclama  process? 

_ 13,  To  what  extent  should 

grades  be  used  for  determining 
distinguished  graduates? 

_ 74.  To  what  extent  have  you 

found  the  ACSC  curriculum  CD- 
ROM  to  be  useful? 

Please  answer  the  following 
questions  by  placing  a  check 
mark  in  the  space  that  most 
describes  your  opinion  about 
that  item. 

75.  What  did  you  think  about  the 
amount  of  assigned  reading? 

_ Too  Much _ Not  enough 

About  right 

16,  What  did  you  think  about  the 

length  of  the  courses? _ ^Too 

Long _ Too  Short  About 

right 

77,  What  did  you  think  about  the 
amount  of  time  allotted  to 
research? 

_ Too  Much  Not  enough 

About  right 


78.  What  did  you  think  about  the 
balance  between  civilian  and 
military  faculty  members? 

More  military  _ More 

civilian  About  right 

79.  What  did  you  think  about  the 
time  allotted  for  student/faculty 
interaction? 

_ ^Too  Much  Not  enough 

About  right 

80.  What  grading  method  do  you 
think  is  more  appropriate  at 
ACSC? 

Letter  grades 
Pass/Fail 


81.  Ifyou  use  the  CD-ROM, 

where  do  you  use  it? _ ^Home 

Work  Don’t  use  it 


D.  Demographic  Data 

Specific  Directions:  This 
information  is  needed  for 
statistical  control  only.  Please 
place  a  check  mark  next  to  the 
response  that  best  describes  the 
category  to  which  you  belong. 

82.  Age: _ 35  and  under 

_ 36-39 

_ ^40  and  over 

_ Other  (please 

specify) _ 

83.  Gender: _ ^Male 

_ Female 

84.  Race: 

African-American 

Asian 

Hispanic 

_ ^Native-American 

White,  non-Hispanic 
_ ^Other  (please  specify) 


85.  Career  Field: 

_ Operations 

(please  specify) _ 

_ Mission  Support 

(please  specify) _ 


Logistics 

(please  specify) _ 

_ ^Medical/Legal 

(please  specify) _ 

_ Other 

(please  specify) _ 

86.  Masters  Degree 
Concentration: 
Humanities 

(please  specify) _ 

_ Social  Sciences 

(please  specify) _ 

_ ^Math  and  Science 

(please  specify) _ 

_ ^Engineering 

(please  specify) _ 

_ ^Medical/Legal 

(please  specify) _ 

_ ^Other 

(please  specify) _ 

No  Master’s  Degree 

_ ^Masters  Plus 

_ Ph.D. 


87,  Commission  Source: 

_ ^ROTC 

_ ^USAFA 

_ OTS 

_ Other 

(please  specify) _ 

£.  Student  Expectations 

Specific  Directions:  This 
information  concerns  your 
expectations  about  ACSC  before 
you  attended  the  program. 
Complete  the  following  sentence 
(check  all  that  apply). 

88.  When  1  found  out  that  1  was 
selected  to  attend  ACSC,  I 
expected  that: 

ACSC  would  improve  my 
chances  for  fiiture  promotions, 
ACSC’s  curriculum  would 
make  me  a  better  Air  Force  officer. 
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_ I  would  meet  other  people 

and  learn  about  other  career  fields. 

_ I  would  improve  my  golf 

game. 

_ would  get  to  spend  more 

time  with  my  family. 

_ I  would  spend  a  great  deal  of 

time  socializing. 

_ would  learn  very  little 

while  a  student  at  ACSC. 

_ would  be  academically 

challenged  while  a  student  at 
ACSC. 

_ ^Other  (please  specify) 


90.  Please  use  this  space  to 
comment  on  any  of  the  items  on 
this  questionnaire.  Be  sure  to 
specify  to  which  item  you  are 
referring. 


F.  Open-Ended  Questions 

Specific  Directions:  The  following 
items  are  open-ended  questions. 
Use  the  back  of  this  form  if 
necessary  but  try  to  direct  your 
comments  to  specific  areas  of  the 
program. 

89.  In  one  or  two  sentences, 
describe  how  you  would  improve 
Air  Command  and  Staff  CoUege. 
Please  be  specific! 


If  you  would  like  to  receive  the 
results  from  this  study,  please 
put  your  name  and  address  in 
the  space  provided  below. 


THANK  YOU! 

AU  SCN  95-1 1  (expires  18  July 
1996) 
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Brenda  F.  Roth 

601  Shepards  Hill  Road 
Charlottesville,  VA  22903 


3  November  1995 

<Title>  <First  Name>  <Last  Name> 

<Addressl> 

<Address2> 

<City>  <State>  <Postal  Code> 

Dear  <Title>  <Last  Name> 

As  a  recent  graduate  of  Air  Command  and  Staff  College,  you  are  in  a  position  to 
evaluate  the  overall  effectiveness  of  the  program  and  to  rate  the  quality  of  ACSC’s 
teaching  methods  and  program  activities.  This  is  important  as  ACSC  continues  to  refine  its 
program  and  the  Air  Force  places  more  emphasis  on  professional  military  education. 

I  am  an  active  duty  Air  Force  officer  pursuing  my  Ph.D.  in  Higher  Education  at  the 
University  of  Virginia.  My  research  involves  a  comprehensive  evaluation  of  ACSC’s 
teaching  methods  and  program  activities.  Your  perceptions  of  the  effectiveness  of  the 
program  are  invaluable  and  will  be  used  to  enhance  the  quality  of  ACSC’s  programs. 
Although  your  participation  in  this  study  is  voluntary,  in  order  that  the  results  of  this  study 
truly  represent  the  perceptions  of  ACSC  graduates,  it  is  important  that  each  questionnaire 
be  completed  and  returned  in  the  enclosed  self-addressed  stamped  envelope.  It  should 
take  you  only  1 5  to  20  minutes  to  complete  the  questionnaire. 

You  may  be  assured  of  complete  confidentiality.  The  questionnaire  has  an 
identification  number  for  mailing  purposes  only.  This  is  so  that  I  may  check  your  name  off 
the  mailing  list  when  your  questionnaire  is  returned.  Your  name  will  never  be  placed  on 
the  questionnaire  itself  and  the  results  of  the  study  will  be  reported  in  aggregate  form  only. 
In  accordance  with  AFI  37-132  and  the  Privacy  Act  of  1974,  the  information  you  provide 
will  be  used  for  official  purposes  only. 

I  would  be  happy  to  answer  any  questions  you  may  have  about  the  study.  Please  write 
or  call  me  at  1-800-531-9748. 

Thank  you  very  much  for  your  assistance. 
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Brenda  F.  Roth 

601  Shepards  Hill  Rd. 
Charlottesville,  VA  22903 


30  November  1995 

«Title»  «FirstName»  «LastName» 

«Addressl» 

«Address_2» 

«City»,  «State»  «PostalCode» 

Dear  «Title»  «LastName» 

About  a  month  ago,  I  wrote  to  you  seeking  your  perceptions  about  the  overall 
effectiveness  of  Air  Command  and  Staff  College  and  asking  you  to  rate  the  quality  of 
ACSC’s  teaching  methods  and  program  activities.  As  of  today,  I  have  not  received  your 
completed  questionnaire.  I  realize  that  you  may  not  have  had  time  to  fill  out  the 
questionnaire  but  I  would  really  like  to  have  your  inputs.  If  you  have  already  retiuned  the 
questionnaire,  I  sincerely  appreciate  your  interest  and  participation. 

The  study  is  being  conducted  as  part  of  my  Ph.D.  work  in  Higher  Education  at  the 
University  of  Virginia  but  it  is  also  an  important  study  to  ACSC  and  to  the  future  of 
professional  military  education  in  the  Air  Force.  I  am  writing  to  you  again  because  the 
study’s  usefulness  depends  on  my  receiving  a  questionnaire  fi’om  each  graduate  of  the 
program.  In  order  for  information  fi-om  the  study  to  be  truly  representative  of  the 
perceptions  of  ACSC  graduates,  it  is  essential  that  each  person  in  the  population  return 
their  questionnaire. 

In  the  event  that  your  questionnaire  has  been  misplaced,  I  am  enclosing  a  replacement. 
I  would  be  happy  to  answer  any  questions  you  have  about  the  study.  Please  write  or  call 
meat  1-800-531-9748. 

Thank  you  very  much  for  your  participation. 
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Last  week,  a  questionnaire  seeking  your  perceptions  about  the  overall  effectiveness  of  Air 
Command  and  Staff  College  was  mailed  to  you.  As  an  active  duty  Air  Force  officer,  your  opinions 
are  invaluable  in  pinpointing  areas  of  the  College  that  should  be  changed  or  modified. 

If  you  have  already  completed  and  returned  the  questionnaire  to  me,  please  accept  my  sincere 
thanks  for  your  participation.  If  not,  please  take  a  few  minutes  to  fill  out  the  questionnaire  today.  I 
am  especially  gratefiil  to  you  for  your  help  because  I  believe  that  your  response  will  be  very  useful  to 
senior  leaders  at  the  College  as  they  continue  to  refine  their  program. 

If  you  did  not  receive  a  questionnaire,  or  if  it  was  misplaced,  please  call  me  at  1-800-531-9748  and 
I  will  get  another  one  in  the  mail  to  you  today. 


APPENDIX  B:  Course  Descriptions 
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Course  Descriptions 

Campaign  Introduction  (OV  500) 

This  course  provides  students  vsith  an  overview  of  the  concepts  and  principles  they  wiU 
encounter  throughout  the  academic  year.  It  is  capstoned  by  a  simulation  which  measures 
student  aptitude  in  all  of  the  succeeding  courses  (20  contact  hours). 

Command  and  Leadership  (CL  500) 

This  course  enhances  student  skills  in  five  areas:  Command  Essential  Skills, 
Communication  Skills,  Quality  Concepts,  Joint  Operations,  and  Computer  Skills.  It 
provides  a  solid  foimdation  for  subsequent  instruction  and  prepares  the  students  for  future 
command  or  staff  positions.  Command  Essential  Skills  introduces  the  resources  available 
to  the  commander  and  provides  various  leadership  techniques.  Some  of  those  skills 
include  understanding  of  critical  Air  Force  positions  on  diversity,  accountability,  and 
homosexuality.  In  the  Communication  Skills  area,  speaking,  writing,  and  research  skills 
are  honed.  Quality  Concepts  introduces  the  basic  terms  and  concepts  key  to  a  Quality  Air 
Force.  In  tomorrow’s  world,  force  employment  will  likely  involve  some  form  of  joint  or 
combined  operation.  The  Joint  Operations  area  begins  presenting  seminars  needed  for 
certification  as  Joint  Professional  Military  Education-Phase  I  graduates.  Finally,  current 
ACSC  computer  equipment  capabilities  are  introduced  as  well  as  the  future  uses  of  the 
computer  to  the  air  campaigner  (55  contact  hours). 

War  Objectives  (WC  500) 

This  course  sets  the  stage  for  the  remainder  of  the  curriculum  by  introducing  and  defining 
the  concepts  essential  to  the  study  of  the  operational  art  of  war  and  by  clarifying  the 
distinction  between  war,  conflict,  and  conflict  termination.  The  course  assumes  war, 
conflict  and  conflict  termination  are  interdependent  elements,  whose  nature  and  meaning 
can  only  be  understood  fully  as  part  of  the  social  and  cultural  context  in  which  they  occur. 
On  the  basis  of  this  view,  a  number  of  conflict  and  conflict  termination  models  are 
developed  through  an  examination  of  the  actors  in,  and  the  motives  and  levels  of  violent 
confi'ontation  (26  contact  hours). 

Military  Theory  (MT  500) 

This  course  looks  at  warfere  systematically.  Modem  warfere  is  an  intellectual  and  a 
technological  phenomenon,  and  military  theorists  have  long  attempted  to  impose  order 
and  rationality  on  what  is  essentially  an  irrational  enterprise.  The  reformation  of  military 
theory  and  the  creation  of  new  paradigms  are  the  first  steps  in  integrating  new  technology 
into  war  fighting.  Discussing  the  strengths,  weaknesses,  uses,  and  relevance  of  such 
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attempts,  from  Sun  Tzu  to  the  most  recent  aerospace  thinkers,  provides  the  students  with 
the  analytical  tools  necessary  to  develop  military  theory  into  the  twenty-first  century  (48 
contact  hours). 

Strategic  Structures  (SS  500) 

This  course  teaches  coalition  theory  and  introduces  power  projection  instruments.  It 
begins  the  process  of  making  security  assessments  and  analyses  of  hostile  and  fiiendly 
centers  of  gravity  as  well  as  the  role  of  intelligence  in  the  National  Security  Process.  This 
course  gives  students  the  opportunity  to  look  at  the  basic  civil/military  leadership  power 
relationships  in  state  and  non-state  entities.  The  students  are  introduced  to  illustrative 
case  studies  as  examples  of  centers  of  gravity  (87  contact  hours). 

Operational  Structures  (OS  500) 

This  course  is  designed  to  give  the  student  a  fimdamental  imderstanding  of  the  military 
instrument  of  national  power  required  for  campaign  planning  in  later  blocks  of  instruction. 
The  course  begins  by  building  a  strong  theoretical  imderstanding  of  objectives,  strategy, 
and  doctrine.  The  student  is  prepared  for  combat  planning  in  the  joint  environment  by 
lessons  on  individual  service  force  structure,  doctrine,  and  force  application.  Basic 
principles  of  logistics,  and  command  and  control  (C2)  are  introduced  followed  by  a  review 
of  current  US  logistics,  intelligence  and  C2  capabilities  and  challenges.  The  course 
concludes  with  an  examination  of  the  use  of  systems  analysis  to  find  centers  of  gravity  of 
military  forces  (90  contact  hours). 

Joint  Operations  and  Campaign  Concepts  (JO  500) 

This  course  introduces  Basic  Service  Joint  Doctrine.  After  studying  the  Joint  Deliberate 
and  Crisis  Action  Planning  System,  this  block  leads  the  student  to  begin  selecting 
campaign  options.  It  also  provides  the  opportunity  to  begin  developing  courses  of  action 
for  traditional  warfere  as  well  as  military  options  short  of  war  (67  contact  hours). 

Air  Campaign  (AC  500) 

This  course  explores  the  military  technical  revolution  (MTR)  and  its  effects  on  warfare.  It 
develops  an  appreciation  of  the  synergistic  contributions  of  air  power  to  the  combat 
commander’s  campaign  plan.  It  lays  the  basis  for  mastering  Operational  Art  in  Air  and 
Space  and  for  the  exploitation  of  air  power  in  support  of  US  national  objectives.  Its  goal 
is  an  imderstanding  of  the  Master  Attack  Plan  from  which  the  Air  Tasking  Order  is 
derived  (136  contact  hours). 
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War  Termination  (WT  500) 

This  course  explores  the  war  termination  concept  beyond  describing  the  moment  marking 
the  war’s  end.  The  students  will  understand  the  campaign  termination  missions  conducted 
by  US  military  forces  in  the  transition  to  peace.  A  phase  of  military  operations,  campaign 
termination  must  be  planned  in  full  coordination  with  war  fighting  operations.  To  this 
end,  the  students  analyze  case  studies  highlighting  the  importance  of  matching  the  ends 
desired  to  the  means  used  in  the  campaign  (21  contact  hours). 

Campaign  2000+  (FC  500) 

This  course  defines  possible  future  force  structures  needed  to  meet  an  imdefined  and 
technologically  accelerating  future  in  the  hope  of  defining  rather  than  reacting  to  change. 
Historical  lessons  are  used  to  forecast  future  trends.  Policy,  resource  allocation, 
acquisition  employment  and  power  projection  issues  are  applied  through  a  final  exam 
which  connect  the  present  to  the  forecasted  future  in  a  series  of  steps  needed  to  shape  and 
prepare  the  military. 


(Source:  Air  Command  and  Staff  College  A  Y96  Curriculum  Plan,  March  1995.) 
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APPENDIX  C:  Percentages  of  Respondents  Scoring  High  (an  average  of  4  and  5) 
on  Outcome  Variables  by  Demographic  Variables  and  Student  Expectations 


Table  C-l  percentages  of  Resoondents 


35  &  under 
n  =  54 


Sconne  High  (4  and  5)  on  Outcome  Vanables  bv  Age 


Rate  (%)  Amon 


36-39  40  &  over 

n=115  n=51 


Joint  Camoai 


35 


36 


37 


38 


39 


40 


41 


42 


43 


44 


45 


Air/Snace  Power 


46 


47 


48 


49 


Command  &  Lead 


5 


6 


12 


13 


14 


15 


16 


Critical  Thinkin 


43  %  f23 


46%  (25 


46  %  (25 


39  %  (21 


43  %  (23 


65  %  (35 


61  %  (33 


69  %  (3 


67  %  (36 


70  %  (38 


63  %  (34 


80  %  (43 


81  %  (44 


57%  (31 


76  %  (41 


43  %  (23 


68  %  (3 


35%  (19 


85  %  (46 


74  %  (40 


33%  (18 


80  %  (43 


87  %  (4 


56  %  (30 


85  %  (46 


65  %  (35 


19%  (10 


66  %  (76 


75  %  (86 


76  %  (8 


62  %  (71 


69  %  (79 


77  %  (88 


76  %  (8 


74  %  (85 


75  %  (86 


86  %  (99 


75  %  (86 


88%  (101 


88  %  (101 


78  %  (90 


88%  (101 


43  %  (50 


66  %  (76 


34  %  (39 


82  %  (94 


62  %  (71 


37  %  (42 


68  %  (78 


81  %  (93 


78  %  (90 


85  %  (98 


75  %  (86 


27  %  (31 


55  %  (28 


59  %  (30 


65  %  (33 


43  %  (22 


51  %  (26 


67  %  (34 


69  %  (35 


63  %  (32 


71  %  (36 


75  %  (38 


67  %  (34 


76  %  (39 


65  %  (33 


71  %  (36 


45  %  (23 


61  %(31 


53  %  (2 


75  %  (38 


73  %  (3 


25  %  (13 


73  %  (37) 


86  %  (44 


73  %  (37 


82  %  (42) 


67  %  (34 


24%  (12 


76  %  (41) 


59  %  (32 


69  %  (3 


61  %(33 


72  %  (39 


63  %  (34) 


76  %  (41 


70  %  (38 


59  %  (32) 


69  %  (37) 


78  %  (42 


63  %  (34 
65  %  (35 


80  %  (43 


89  %  (48 


59  %  (32 


61  %  (33 


65  %  (35 


78  %  (90 


70  %  (81 


83  %  (95 


67  %  (7 


75  %  (86 


84  %  (97) 


84  %  (97) 


80  %  (92) 


70  %  (81 


81  %  (93 


83  %  (95 


81  %  (93 
70  %  (81 


88%  (101) 


95  %  (109 


70  %  (80 


70  %  (80 


83  %  (95 


63  %  (32 


51  %  (26 


71  %  (36) 


53  %  (27 


71  %  (36) 


78  %  (40 


76  %  (39 


71  %  (36 


59  %  (30 


67  %  (34 


69  %  (35 


67  %  (34 


61  %(31 


90  %  (46 


88  %  (45 


43  %  (22 


63  %  (32 


67  %  (34 


Table  C-2  : 


m 


Percentages  of  ResDondents  Scorine  Hi 


Rate  (Vo) 


4  and  5)  on  Outcome  Variables  bv  Gender 


Amon 


Item 

Male 

Female 

n=179 

n=41 

Joint  Camnai 


35 


36 


37 


38 


39 


40 


41 


42 


43 


44 


45 


Air/Snace  Power 


46 


47 


48 


49 


Command  &  Lead 


5 


6 


8 


9 


10 


11 


12 


13 


14 


15 


16 


Critical  Thinkin 


17 


58%fl04 


63%ril2 


66%ril8 


49  %  m 


58%  (104 


72%  (129 


68  %  (122 


69  %  (123 


69%  023 


78  %  (139 


68  %  (122 


82%  (14 


84%  (151 


72  %  (129 


80%  (143 


44  %  (78 


65%  (116 


37  %  (6 


82%  (146 
67%  (120 


33  %  (59 


76%  (136 


85%  (152 


74%  (132 


85  %  (152 


69  %  (124 


23  %  (41 


75%  (135 


65%  (116 


77%  (138 


64%  (114 


72%  (129 


77%  (138 


80  %  (144 


75%  (134 


83  %  (149 


75%  (135 


78  %  (139) 


73%  (130 


68%  (122 


86%  (154) 


93%  (166 


63%  (113 


64%  (115 


75%  (134 


56  %  (23 


71  %  (29 


66  %  (2 


63  %  (26 


59  %  (24 


68  %  (28 


80  %  (33 


76  %  (31 


85  %  (35 


88  %  (36 


78  %  (32 


83  %  (34 


80  %  (33 


61  %  (25 


85  %  (35 


44%  (18 


68  %  (28 


44%  (18 


78  %  (32 


68  %  (28 


34  %  (14) 


54  %  (22 


78  %  (32 
63  %  (26 


83  %  (34 


76  %  (31 


29%  (12 


68  %  (28 


56  %  (23 
73  %  (30 


56  %  (23 


78  %  (32 


80  %  (33) 


80  %  (33 


78  %  (32 


63  %  (26 


71  %  (29 


80  %  (33 


76  %  (31 


61  %  (25 


88  %  (36 


88  %  (36 


51  %(21 


73  %  (30 


73  %  (30 
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Table  C-3 : 


^rcenta^sofRe|^d^s_S^Og£S^^j^iJ25=^^^=i=^^====^^= 


Rate  (%)  Among  I 

Item 

African-Am 

Asian 

Hispanic 

Native-Am 

n=17 

n=4 

n=10 

n=4 

IBHHHi 

bhbbi 

■■■■IIH 

35 

■QQBSH 

IBEBBI 

^BBSB 

BrlHIIMi 

■.iiamHM 

36 

BBBB 

b^bbi 

bebsb 

B.iliifafSB 

37 

msmBm 

bbbbi 

38 

ib^ebeb 

^BBSB 

bsbbi 

^^BlEEiB 

39 

HBBSBI 

bsbsb 

BddiRUilsEB 

40 

bcbsb 

^b^iebi 

BiliTH 

41 

■QIESR5HI 

BEEBEiB 

beebeeb 

42 

■dsbqb 

^^BOB 

bdebeib 

43 

besbqb 

BEEBiEB 

beebeeb 

44 

■qqesb 

BbltliMUtiM! 

beeebeeb 

liiMieilB 

45 

bbbbi 

BEEBESB 

BiVMHmB 

46 

^eebesb 

BDSEBE9B 

BBIIESB 

47 

bbbbi 

^EEBESB 

bbbbi 

B:fcl!MiiE}AB 

48 

■esesrebi 

bbbb 

BB^BB 

beebeeb 

BSBRSB 

49 

■^^mB 

hbsb 

bdebeib 

B:>-lBHgfBi 

Command  &  Lead 

5 

^BBRB 

^bbbb 

44  %  (76) 

6 

■ESBinB 

WBSE3SM 

BEEBEBI 

beebebi 

BdclMilEMI 

7 

BismsB 

b^bbhi 

B^^IBB 

^bbsh 

8 

^^bieib 

besbsb 

hbsb 

BiliBibB 

9 

B^^nOB 

BiliBtiB 

Brftliy 

B.-ifAMifcfeiB 

10 

kibbb 

^bbsbi 

Bj«imB 

75  %  (3) 

BEBBB 

11 

^^BEESB 

bbbi 

75  %  (3) 

12 

^ElB^B 

bbbsb 

B[«r(faB 

h^bi 

BaBmB 

13 

besbebi 

B»1«fcii1<iliB 

bdsebeeb 

HBi^B 

14 

BfTTTOB 

IBEBEBI 

bbbeeb 

B:llMliliM 

15 

mmsm 

BfniTTIl’TBI 

16 

Bll  IIIB 

bebebi 

^EEBQB 

50  %  (2) 

^bbzbb 

Critical  Thinking 

bbbbi 

bbbi^b 

■■■■■■ 

17 

76%  (13) 

75  %  (3) 

80  %  (8) 

beebeeb 

Bril'iTIMEB 

18 

■qEBB 

biebsb 

^^BOB 

bdeebesb 

BAIjMimB 

19 

BBXERB 

HBRB 

100%  (4) 

BfirMHtoM 

20 

BFtiHfiB 

Brn^TEB 

bbbb 

bbbeeb 

Kimins 

21 

■E^^[EB 

^EEBEIB 

^bbsb 

mi^mm 

22 

beebsb 

^EEBESB 

bdiebeeb 

Br/l'TItriB 

23 

b^rsb 

bieebesb 

beebeeb 

beebeeb 

■BBBB 

24 

76%  (13) 

beejbsb 

^EEBESB 

■^^beb 

BrftiTifT^B 

25 

H^BBB 

beebebi 

70  %  (7) 

75  %  (3) 

BAMWIKB 

26 

■rnCTnat 

beebebi 

beebebi 

bbbeeb 

■QBESEB 

27 

beebebi 

beebbb 

100  %  (4) 

BITTBfnB 

28 

BiiEimnB 

■eebbi 

bbbbh 

BOEEBEEB 

BfiilBiifcUB 

29 

■EBRBB 

^E^mB 

beebebi 

beebeeb 

BEBSSB 

30 

82%  (14) 

■dse^eb 

BESEffIliB 

■bbeeb 

86%  (155)  1 

31 

94  %  (16) 

bebesb 

100%  (10) 

bbebeeb 

B*fi!HfTf™ 

32 

bbbbb 

■i^beb 

bbbbi 

BEiBBl 

33 

i^^eessb 

beebebi 

b^bbi 

bbbeb 

34 

88%  (15) 

60  %  (6) 

Br/tlMI 

BritMllklM 

Table  C-4  :  Percentages  of  Restx)ndents 


Operations 

n=82 


Scoring  High  (4  and  5)  on  Outcome  Variables  bv  Career  Field 


Rate  f%)  Amon 


Support  Logistics  Med/Legal 


Joint  Camoai 


35 


36 


37 


38 


39 


40 


41 


42 


43 


44 


45 


Air/Soace  Power 


46 


47 


48 


49 


Command  &  Lead 


5 


6 


8 


9 


10 


11 


12 


13 


14 


15 


16 


Critical  Thinkin 


17 


18 


19 


20 


21 


22 


23 


24 


25 


26 


27 


28 


29 


30 


31 


32 


33 


34 


52  %  r43 


60  %  f49 


61  %  f50 


46  %  f38 


57  %  (4 


71  %  (58 


71  %  (58 


76  %  (62 


74  %  (61 


78  %  (640 


61  %  (50 


83  %  (68 


85  %  (70 


76  %  (62 


84  %  (69) 


41  %  (34 


61  %  (50 


30  %  (25 


80  %  (66 


65  %  (53 


30  %  (25 


72  %  (59 


83  %  (68 


67%  (55) 


85%(70 


70  %  (57) 


27  %  (22 


59  %  (44 


65  %  (49 


69  %  (52 


56  %  (42 


55  %  (41 


65  %  (49 


64  %  (48 


60  %  (45 


67  %  (50 


76  %  (5 


69  %  (52 


79  %  (59 


77  %  (58 


68  %  (51 


77  %  (58 


47  %  (35 


68  %  (51 


40  %  (30 


83  %  (62 


75  %  (56 


36  %  (2 


69  %  (52 


83  %  (62) 


77  %  (58 


80  %  (60 


21  %(16 


55  %  (23 


64  %  (2 


64  %  (2 


50  %  (21) 


62  %  (26 


74  %  (31 


76  %  (32 


67  %  (28 


69%  (29 


83  %  (35 


81  %  (34 


81  %  (34 


88  %  (3 


55  %  (23 


74  %  (31 


43  %  (180 


74  %  (31 


48  ®/H20) 


81  %  (34 


60  %  (25 


29%  (12 


71%  (30 


83% 


64  %  (27) 


90  %  (38 


69%  (29 


24%  (10 


86  %  (6 


71  %  (5 


71  %  (5 


86  %  (6 


100% 


100% 


86  %  (6 


100% 


100% 


29  %  (2 


57  %  (4 


57%  (4 


86  %  (6 


71  %  (5 


86  %  (6 


86  %  (6 


86  %  (6) 


100  % 


86  %  (6 


70  %  (57) 


61  %(50) 


72  %  (59 


59  %  (48 


82  %  (6 


70  %  (5 


67  %  (55 


76  %  (62 


87  %  (71 


76  %  (62) 


71%  (58 


83%  (68 


93%  (76 


65%  (53 


68  %  (56 


73  %  (55 


56  %  (42 


79  %  (59 


56  %  (42 


68  %  (51 
77  %  (58 


77  %  (58 


73%  (55 


60  %  (45 


71%  (53 


73%  (55 


69  %  (52 


55%  (41 


79  %  (33) 


74%  (31) 


79  %  (33 


74  %  (31 


79  %  (33 


81%  (34 


79  %  (33 


81%  (34 


57  %  (24 


71%  (30 


71%  (30 


67  %  (28 


69  %  (29 


81%  (34 


71  %  (5 


100% 


71  %  (5 


91%  (68 
52%  (38 


61  %  (46 


67  %  (50; 


86  %  (6 


86  %  (6 


100  % 


100  % 
100  % 


86  %  6 


100% 


100  % 


100% 


100  % 


67  %  (28 


62  %  (26 


76  %  (32) 


100% 


86  %  (6 


218 


Table  C-5 : 


Rate  (%)  Among 

_ _ 1 

Item 

Humanities 
n  =  ll 

Soc  Science 
n=108 

Math/Science 
n  =  52 

Engineering 
n  =  34 

Med/Legal 

n=6 

Joint  Campaign 

35 

55  %  (6) 

59%  (64) 

56  %  (29) 

bbbsb 

36 

45%  (5) 

68  %  (73) 

71%  (37) 

EBl^EBfli 

67  %  (4)  1 

37 

■EE^^I 

■ISEED3H 

^^^EoBI 

38 

■ULMSiH 

b^ie^h 

KS^RmB 

eeebesb 

39 

64%  (7) 

e^sesb 

eeb^ri^bi 

40 

BIB9B 

■EBB 

41 

74%  (80) 

BUsMkB 

bes^e^bi 

bb^bb 

42 

Mill  HIM 

bes^bh 

43 

■BESIESH 

74%  (80) 

■eebh 

44 

■iMSini 

81%  (42) 

ibbsssb 

83%  (5)  1 

45 

■^^esih 

Air/Space  Power 

46 

KkKB 

BESi^^B 

47 

msmm 

48 

EI^^S9B 

100%  (6)  1 

49 

eesssesei 

■bhesh 

Command  &  Lead 

1 

5 

BBBB 

6 

■I^EB 

^^@B 

7 

Bi^nmE 

8 

■mim 

bdeseesbi 

BBmB 

9 

■smiH 

^EmSSB 

^hb 

10 

35%  918) 

BmB 

11 

I^^EEB 

^hb 

12 

91%  (10) 

[i^^nuEi 

KkIffiskB 

BESBSB 

13 

■.L-UlbM 

■e^eeh 

EBl^mHI 

Bm<E£H 

14 

83%  (43) 

BEBSB 

15 

iEmnEn 

71%  (77) 

67%  (35) 

65%  (22) 

^kbb 

16 

BBSB 

BBIB 

Critical  Thinking 

bhhii^i 

17 

64%  (7) 

79%  (41) 

BimEBi 

B^BSB 

18 

64%  (7) 

BBSBBi 

19 

64%  (7) 

81%  (88) 

75%  (39) 

81%  (26) 

BUBIEB 

20 

■mH 

Elg^ESH 

■ebseh 

^miUB 

21 

■EIS3EI3Hi 

■BEIBI 

22 

^EiEEmB 

BM^mB 

23 

73%  (8) 

79%  (85) 

88%  (46) 

h^bhi 

w^msm 

24 

79%  (41) 

^e^^h 

BEE^^B 

25 

■Bm 

■SESE^H 

BEQ^mB 

26 

71%  (77) 

boiex^bi 

BEBSB 

27 

EBSSESiH 

^■esesb 

28 

64%  (7) 

IBnESEH 

B^OB 

EEEIBHi 

BEMS&B 

29 

■Hn 

69%  (74) 

BDE^^B 

30 

B^RSB 

be^ebi 

31 

■essie^sh 

B^@B 

94%  (30) 

■eebh 

32 

WiilftisB 

bebeqb 

33 

■Hn 

bbebib 

bbe^sh 

34 

■on 

83%  (43) 

bbexese 

■eeebi 

9 


Table  C-6  :  Percentages  of  Respondents  Scoring  High  (A  and  5)  on  Outcome  Variables  bv  Education 


Rate  (%)  Amon 


Masters  Masters  + 

n=211  n=16 


Joint  Camnai 


35 


36 


37 


38 


39 


40 


41 


42 


43 


44 


45 


Air/Soace  Power 


46 


47 


48 


49 


Command  &  Lead 


5 


6 


8 


9 


10 


11 


12 


13 


14 


15 


16 


Cntical  Thinkin 


17 


18 


19 


20 


21 


22 


23 


24 


25 


26 


27 


28 


29 


30 


31 


32 


33 


34 


75  %  f3 


25%ri 


25  %n 


50  %  (2 


75  %  (I 


50  %  f2 


50  %  (2 


75  %  f3 


57%  021 


64%  034 


66%  039 


53%  011 


58%  023 


73  %  053 


72  %  052 


71  %  050 


74%  056 


80  %  068 


68  %  044 


75  %  02 


44% 


25  %  (A 


25  %  (A 


38% 


69%  01 


69%ril 


75%  02 


75  %  02 


75%  02 


56  %  ^9 


Ph.  D. 
n=13 


62  %  r8 


54% 


46  %  r6 


31  %r4 


38  %  f5 


92  %  02 


62  %  r8 


77%O0 


38  %  r5 


100  %  f4 


50  %  f2 


50  %  r2 


25%fl 


50  %  a 


50  %  f2 


50  %  a 


75%  (3) 


84%  07 


84%  07 


72%  052 


82%  074 


44  %  f92 


66  %  040 


38%r81 


82  %  073 


68%  044 


32  %  (^6 


72%  052 


84  %  078 


72%  051 


85%  079 


71  %  049 


81  %  f  13 


75%  02 


75%  02 


81%  03 


56  %  f9 


75%  02 


50  %  r8 


94%  05 


63%  OO 


44% 


88  %  04 


94%  05 


75%  02 


94%  05 


81%  03 


77%O0 


77%O0 


62  %  f8 


92%  02 


38% 


31  %r4 


23% 


46  %  (6 


3 


62 


77%  OO 


77%  OO 


69  %  (9 


54% 


75%  058 


64%  036 


79%  066 


63%  (133 


73  %  (154 


78%  065 


80  %  069 


75%  059 


66%  040 


77%  (162 


81  %O70 


75%  (159 


67%  (142 


88%  085 


92  %  095 


63  %  (132 


69  %  046 


75%  (158 


88  %  04 


75%  02 


81%  03 


81%  03 


63%  (10 


69%  01 


81%  03 


75%  02 


75%  (12 


88%  04 


75%  02 


81%  (13 


81%  03 


81%  03 


81%  (13 


63%  OO 


81%  (13 


77%  (10 


77%  (10 


85%  (11 


54% 


77%  OO 


92%  02 


92%  (12 


92%  02 


92%  02 


92%  (12 


77%  (10 


92%  02 


69  %  (9 


77%  (10 


85%  01 


92%  02 


Table  C-7 :  Percentaees 


Commission  Source 


of  Respondents  Scoring  High  (4  and  5)  on  Outcome  Variables  b 


ROTC 

n=91 


Rate  (%)  Amon 


USAFA 

n==44 


Joint  Camoai 


35 


36 


37 


38 


39 


40 


41 


42 


43 


44 


45 


Air/Space  Power 


46 


47 


48 


49 


Command  &  Lead 


5 


6 


8 


9 


10 


11 


12 


13 


14 


15 


16 


Critical  Thinkin 


17 


18 


19 


20 


21 


22 


23 


24 


25 


26 


27 


28 


29 


30 


31 


32 


33 


3 


57  %  (52 


66  %  (60 


65%  (59 


49%  (45 


62%  (56 


69%  (63 


75%  (68 


70%  (64 


73%  (66 


79%  (72 


69%  (63 


48%  (21 


61%  (27 


55%  (24 


59%  (26 


70%  (31 


66%  (29 


75%  (33 


68%  ( 


60  %  (4 


63%  (49 


69%  (54 


54%  (42 


55%  (43 


72%  (56' 


69%  (54 


68%  (53 


67%  (52 


74%  (58 


71%  (55 


82%  (75 


85% 


70%  (64 


81%  (74 


42%  (38 


64%  (58 


37%  (34 


68%  (30 


82%  (36 


43%  (19 


73%  (32 


30%  (13 


91%  (40 


70%  (64 


38%  (35 


87%  (79 


69%  (63 


86%  (78 


67%  (61 


23% 


32%  (14 


73%  (32 


82%  (36 


68%  (30 


86%  (38 


73%  (32 


25%  (11 


77%  (60 


79%  (62 


67%  (52 


78%  (61 


63%  (49 


42%  (33 


72%  (56 


68%  (53 


27%  (21 


71%  (55 


81%  (63 


76%  (59 


82%  (64 


72%  (56 


26%  (20 


75%  (68 


65%  (59 


77%  (34 


64%  (28 


80%  (35 


77%  (70 


76%  (69 


82%  (75 


69%  (63 


59%  (54 


73%  (66 


69%  (63 


69%  (63 


82%  (75) 


93%  (85 


59%  (54 


67%  (61 


6 


73%  (32 


80%  (35 


80%  (35 


66%  (29 


75%  (33 


89%  (39 


60%  (31 


66%  (29 


86%  (38 


89%  (39 


69%  (54 


59%  (46 


76%  (59 


62%  (48 


82%  (64 


77%  (60 


78%  (61 


67%  (52 


55%  (24 


75%  (33) 


81%  (63 


74%  (58 


67%  (52 


90%  (70 


91%  (71 


59%  (46 


68%  (53 


76%  (59 


Table  C-8:  Percentages  of  Restwndents  Scoring  High  (4  and  5)  on  Outcome  Variables  by  Student 
Expectations _ 


Rate  (%)  Among  | 

Item 

Expectation  1 
n=191 

Expectation  2 
n=190 

Expectation  3 
n=196 

Expectation  4 
n=36 

Joint  Campaign 

35 

56%  (107) 

60%  (114) 

59%  (116) 

53%  (19) 

36 

64%  (123) 

67%  (127) 

67%  (131) 

56%  (20) 

37 

68%  (130) 

69%  (131) 

68%  (134) 

72%  (26) 

38 

52%  (99) 

55%  (105) 

53%  (104) 

56%  (20) 

39 

59%  (113) 

61%  (115) 

61%  (119) 

58%  (21) 

40 

72%  (138) 

74%  (140) 

73%  (143) 

72%  (26) 

41 

71%  (135) 

73%  (139) 

73%  (143) 

67%  (24) 

42 

69%  (132) 

72%  (136) 

71%  (140) 

61%  (22) 

43 

73%  (139) 

74%  (140) 

74%  (146) 

72%  (26) 

44 

81%  (155) 

80%  (152) 

83%  (162) 

83%  (30) 

45 

69%  (132) 

70%  (133) 

73%  (143) 

75%  (27) 

Air/Space  Power 

46 

83%  (158) 

84%  (159) 

84%  (164) 

92%  (33) 

47 

84%  (161) 

86%  (163) 

85%  (166) 

94%  (34) 

48 

70%  (134) 

72%  (136) 

70%  (137) 

75%  (27) 

49 

81%  (155) 

83%  (157) 

83%  (162) 

72%  (26) 

Command  &  Lead 

5 

43  %  (83) 

43%  (81) 

45%  (88) 

47%  (17) 

6 

64%  (123) 

64%  (122) 

66%  (129) 

69%  (25) 

7 

38%  (73) 

37%  (70) 

40%  (79) 

36%  (13) 

8 

81%  (154) 

80%  (152) 

83%  (163) 

86%  (31) 

9 

69%  (131) 

68%  (130) 

68%  (134) 

72%  (26) 

10 

30%  (58) 

33%  (62) 

34%  (66) 

31%  (11) 

11 

71%  (136) 

72%  (137) 

71%  (140) 

81%  (29) 

12 

84%  (160) 

84%  (159) 

85%  (166) 

89%  (32) 

13 

71%  (136) 

73%  (139) 

72%  (142) 

61%  (22) 

14 

84%  (161) 

84%  (160) 

85%  (166) 

89%  (32) 

15 

70%  (133) 

71%  (134) 

72%  (141) 

72%  (26) 

16 

25%  (47) 

25%  (48) 

25%  (49) 

19%  (7) 

Critical  Thinking 

17 

74%  (142) 

73%  (138) 

75%  (147) 

83%  (30) 

18 

65%  (124) 

64%  (122) 

64%  (126) 

69%  (25) 

19 

76%  (146) 

76%  (144) 

78%  (152) 

83%  (30) 

20 

61%  (116) 

63%  (120) 

62%  (121) 

69%  (25) 

21 

74%  (141) 

74%  (140) 

76%  (148) 

61%  (22) 

22 

77%  (147) 

78%  (148) 

80%  (156) 

81%  (29) 

23 

80%  (152) 

79%  (151) 

81%  (159) 

81%  (29) 

24 

76%  (145) 

76%  (144) 

78%  (152) 

75%  (27) 

25 

64%  (123) 

69%  (129) 

68%  (134) 

56%  (20) 

26 

75%  (144) 

74%  (141) 

78%  (152) 

67%  (24) 

27 

78%  (149) 

79%  (150) 

81%  (158) 

81%  (29) 

28 

75%  (143) 

75%  (142) 

76%  (149) 

69%  (25) 

29 

65%  125) 

68%  (130) 

68%  (134) 

67%  (24) 

30 

86%  (165) 

87%  (166) 

87%  (171) 

89%  (32) 

31 

92%  (176) 

92%  (175) 

94%  (184) 

94%  (34) 

32 

59%  (113) 

61%  (116) 

62%  (122) 

58%  (21) 

33 

65%  (125) 

67%  (127) 

68%  (134) 

56%  (20) 

34 

74%  (142) 

75%  (142) 

77%  (151) 

67%  (24) 

222 


Table  C-9  :  Percentages  of  Respondents  Scoring  (4  and  5)  on  Outcome  Variables  by  Student  Expectations 
(continued) 


Rate  (%  )  Among  I 

Item 

Expectation  5 
n=104 

Expectation  6 
n=42 

Expectation  7 
n=6 

Expectation  8 
n=143 

Expectation  9 
n=26 

Joint  Campaign 

hhhhh 

35 

62%  (26) 

58%  (15) 

36 

HmkiE 

62%  (16) 

37 

^KEmSH 

EUiTiM^m 

65%  (17) 

38 

HILIUM 

38%  (10) 

39 

54%  (14) 

40 

^EEE&m 

69%  (18) 

41 

MEPfESHI 

mrUiTtlllM 

65%  (17) 

42 

w^mm 

Mr/iTTH'lri'm 

73%  (19) 

43 

mriliTtH  jUi 

81%  (21) 

44 

Kimmfli 

HBH 

^e^bbse 

92%  (24) 

45 

Hzm&cm 

E^liTflTPi 

81%  (21) 

■■■■■■ 

46 

WESMSBSE^ 

■■LUHUM 

88%  (23) 

47 

ia:<ri>{fili>X11 

88%  (23) 

48 

WBBSSSEBM 

WESSSSSMi 

■beh 

mrMiTtiriW 

73%  (19) 

49 

85%  (22) 

Command  &  Lead 

5 

46%  (12) 

6 

■Emsnsi 

■nsMSH 

69%  (18) 

7 

■i^EiBmii 

46%  (12) 

8 

82%  (85) 

79%  (33) 

67%  (4) 

88%  (23) 

9 

66%  (69) 

69%  (29) 

69%  (18) 

10 

MEmnMi 

■EEmnH 

27%  (7) 

11 

bequh 

96%  (25) 

12 

■ESI^mE 

■iBnmH 

96%  (25) 

13 

EQEE^H 

■QiEmH 

73%  (19) 

14 

HnMlSE 

100%  (26) 

15 

■EnEQESH 

65%  (17) 

16 

23%  (24) 

17%  (7) 

^K^SSIE 

29%  (41) 

15%  (4) 

Critical  Thinking 

17 

74%  (77) 

74%  (31) 

HnSH 

77%  (20) 

18 

64%  (67) 

EESnE^HI 

58%  (15) 

19 

83%  (5) 

73%  (19) 

20 

EEEESSEk 

■Esissmi 

54%  (14) 

21 

61%  (28) 

69%  (18) 

22 

81%  (84) 

■ussmni 

HEMI0E 

iBmomi 

81%  (21) 

23 

■KESI^ISni 

84%  (120) 

81%  (21) 

24 

■SEmH 

WKMESm 

85%  (22) 

25 

ee^smkl 

69%  (18) 

26 

■b^esh 

MiliiiaiLf 

85%  (22) 

27 

■esg^hi 

■DsnEi 

81%  (21) 

28 

■QEmH 

■BoEIEfli 

73%  (19) 

29 

HimmH 

62%  (16) 

30 

86%  (36) 

85%  (22) 

31 

94%  (98) 

■HCSE 

94%  (134) 

96%  (25) 

32 

■EnmHi 

69%  (18) 

33 

63%  (65) 

67%  (28) 

■qexehi 

65%  (17) 

34 

71%  (30) 

j/g^SSBStE 

■ebeqeh 

92%  (24) 
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APPENDIX  D:  Correlations  Between  Environmental  Variables  and  Outcome 
Variables  by  Item 


Significant  at  .01  Level 


Significant  at  .01  Level 


Table  D-5  :  Correlations  Between  Faculty  and  Outcome  Variables  by  Item 


Significant  at  .01  Level 


Table  D-6  :  Correlations 


Item 


Joint  Camnai 


Between  Gradine  and  Outcome  Variables  bv  Item 


Grading  (n=228 


66  71  72  73 


35 

.17 

.22* 

.15 

.12 

36 

.17 

.22* 

.10 

.10 

37 

.21* 

.24* 

.07 

.13 

38 

.15 

.15 

.13 

.08 

39 

.14 

.14 

.14 

.18* 

40 

.15 

.24* 

.10 

.20* 

41 

.15 

.20* 

.06 

.19* 

42 

.16 

.29* 

.18* 

.21* 

43 

.11 

.18* 

.15 

.16 

44 

.04 

.13 

,16 

.19* 

45 

.05 

.10 

.02 

.08 

Air/Snace  Power 


46 


47 


48 


49 


Command  &  Lead 


5 

.02 

.04 

-.03 

-.02 

6 

.08 

.08 

.13 

.02 

7 

.12 

.18* 

.09 

.12 

8 

.05 

.11 

.11 

.08 

9 

.13 

.14 

.11 

.09 

10 

.25* 

.16 

.09 

.08 

11 

.10 

.17 

.16 

.12 

12 

.07 

.10 

.19* 

.14 

13 

.14 

.25* 

.22* 

.03 

14 

.10 

.13 

.11 

.10 

15 

.05 

.11 

.08 

-.01 

16 

.15 

.23* 

.17 

.01 

Critical  Thinkin 


17 

-.06 

-.04 

.08 

18 

.06 

.12 

.20* 

19 

.03 

.06 

.16 

20 

.15 

.11 

.31* 

21 

.09 

.09 

.12 

22 

.01 

.10 

.14 

23 

.06 

.16 

.13 

24 

.10 

.15 

.10 

25 

.09 

.17 

.06 

26 

.12 

.17 

.03 

27 

.12 

.17 

,09 

28 

.11 

.21* 

.11 

29 

-.05 

.09 

-.01 

30 

.10 

00 

.09 

31 

.10 

.18* 

.13 

32 

.03 

.17 

.14 

33 

.15 

.21* 

.18* 

34 

-.01 

.19* 

-.00 

Significant  at  .01  Level 


